2023-12-06T15:16:06.288 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-06T15:16:06.288 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-12-06_15:04:09-rbd-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7480288 --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} --name yuriw-2023-12-06_15:04:09-rbd-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2023-12-06_15:04:09-rbd-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7480288/orig.config.yaml 2023-12-06T15:16:06.335 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-06T15:16:06.415 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-06_15:04:09-rbd-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7480288 branch: wip-yuri6-testing-2023-12-05-0753-pacific description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: null first_in_suite: false job_id: '7480288' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: yuriw-2023-12-06_15:04:09-rbd-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri6-testing-2023-12-05-0753-pacific ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2023-12-05-0753-pacific sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 5062 sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-yuri6-testing-2023-12-05-0753-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b targets: smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdeHQEar7hPqc96kDyNSh6M7LB7BwlUWoPVhBK4v+HkXWC8SO3dk4cEx/FQYQnJlhdNSE30SESLYGC+N2hMAnc= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD4ib3EayXX/EbckMYpTCjmhZ2CmtXTzz/JTEtV7vSVUI06kQZE5WQuceaANryCFJip34iVZ39qXWIo6HE17Fmo= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-06_15:04:09 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-06T15:16:06.415 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa; will attempt to use it 2023-12-06T15:16:06.416 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks 2023-12-06T15:16:06.416 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-06T15:16:06.417 INFO:teuthology.task.internal:Checking packages... 2023-12-06T15:16:06.440 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd03fca4f35d43f61b4679936aef7bd2f01cced0b' 2023-12-06T15:16:06.440 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-06T15:16:06.440 INFO:teuthology.packaging:ref: None 2023-12-06T15:16:06.440 INFO:teuthology.packaging:tag: None 2023-12-06T15:16:06.440 INFO:teuthology.packaging:branch: wip-yuri6-testing-2023-12-05-0753-pacific 2023-12-06T15:16:06.440 INFO:teuthology.packaging:sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T15:16:06.440 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri6-testing-2023-12-05-0753-pacific 2023-12-06T15:16:06.583 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-335.gd03fca4f 2023-12-06T15:16:06.583 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-06T15:16:06.590 INFO:teuthology.task.internal:no buildpackages task found 2023-12-06T15:16:06.590 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-06T15:16:06.604 INFO:teuthology.task.internal:Saving configuration 2023-12-06T15:16:06.623 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-06T15:16:06.630 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-06T15:16:06.658 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-06_15:04:09-rbd-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7480288', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 15:10:53.335776', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdeHQEar7hPqc96kDyNSh6M7LB7BwlUWoPVhBK4v+HkXWC8SO3dk4cEx/FQYQnJlhdNSE30SESLYGC+N2hMAnc='} 2023-12-06T15:16:06.683 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-06_15:04:09-rbd-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7480288', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 15:10:53.333529', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD4ib3EayXX/EbckMYpTCjmhZ2CmtXTzz/JTEtV7vSVUI06kQZE5WQuceaANryCFJip34iVZ39qXWIo6HE17Fmo='} 2023-12-06T15:16:06.683 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-06T15:16:06.689 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-12-06T15:16:06.689 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-12-06T15:16:06.690 INFO:teuthology.run_tasks:Running task console_log... 2023-12-06T15:16:06.791 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2cd2a99f70>, signals=[15]) 2023-12-06T15:16:06.791 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-06T15:16:06.798 INFO:teuthology.task.internal:Opening connections... 2023-12-06T15:16:06.798 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2023-12-06T15:16:06.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T15:16:06.867 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2023-12-06T15:16:06.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T15:16:06.933 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-06T15:16:06.939 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2023-12-06T15:16:06.960 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2023-12-06T15:16:06.961 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2023-12-06T15:16:07.020 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2023-12-06T15:16:07.020 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="8" 2023-12-06T15:16:07.020 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2023-12-06T15:16:07.020 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2023-12-06T15:16:07.021 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="8" 2023-12-06T15:16:07.021 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el8" 2023-12-06T15:16:07.021 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-06T15:16:07.021 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2023-12-06T15:16:07.021 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-06T15:16:07.021 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2023-12-06T15:16:07.021 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-06T15:16:07.021 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-06T15:16:07.021 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-06T15:16:07.022 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2023-12-06T15:16:07.047 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2023-12-06T15:16:07.069 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2023-12-06T15:16:07.069 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2023-12-06T15:16:07.129 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2023-12-06T15:16:07.129 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="8" 2023-12-06T15:16:07.130 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2023-12-06T15:16:07.130 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2023-12-06T15:16:07.130 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="8" 2023-12-06T15:16:07.130 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el8" 2023-12-06T15:16:07.130 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-06T15:16:07.130 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2023-12-06T15:16:07.130 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-06T15:16:07.130 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2023-12-06T15:16:07.130 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-06T15:16:07.130 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-06T15:16:07.130 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-06T15:16:07.131 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2023-12-06T15:16:07.155 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-06T15:16:07.165 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-06T15:16:07.248 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-06T15:16:07.249 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2023-12-06T15:16:07.251 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2023-12-06T15:16:07.274 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-06T15:16:07.280 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-06T15:16:07.280 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2023-12-06T15:16:07.313 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2023-12-06T15:16:07.346 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-06T15:16:07.402 INFO:teuthology.run_tasks:Running task kernel... 2023-12-06T15:16:07.416 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-06T15:16:07.416 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-06T15:16:07.417 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-06T15:16:07.417 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-06T15:16:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:16:07.437 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2023-12-06T15:16:07.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:16:07.438 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2023-12-06T15:16:07.497 INFO:teuthology.orchestra.run.smithi176.stdout:4.18.0-489.el8.x86_64 2023-12-06T15:16:07.497 INFO:teuthology.task.kernel:Running kernel on smithi176: 4.18.0-489.el8.x86_64 2023-12-06T15:16:07.497 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2023-12-06T15:16:07.502 INFO:teuthology.orchestra.run.smithi204.stdout:4.18.0-489.el8.x86_64 2023-12-06T15:16:07.502 INFO:teuthology.task.kernel:Running kernel on smithi204: 4.18.0-489.el8.x86_64 2023-12-06T15:16:07.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum install -y kernel 2023-12-06T15:16:11.584 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - AppStream 22 MB/s | 34 MB 00:01 2023-12-06T15:16:13.916 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - AppStream 9.3 MB/s | 34 MB 00:03 2023-12-06T15:16:22.550 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 55 MB 00:02 2023-12-06T15:16:30.672 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - Extras 54 kB/s | 18 kB 00:00 2023-12-06T15:16:31.358 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-12-06T15:16:31.702 INFO:teuthology.orchestra.run.smithi204.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-12-06T15:16:32.563 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 22 MB/s | 16 MB 00:00 2023-12-06T15:16:36.353 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 603 kB/s | 24 kB 00:00 2023-12-06T15:16:39.994 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-06T15:16:39.994 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-06T15:16:40.197 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - BaseOS 2.5 MB/s | 55 MB 00:22 2023-12-06T15:16:40.299 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:16:40.299 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:16:40.300 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2023-12-06T15:16:40.300 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:16:40.300 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:16:40.300 INFO:teuthology.orchestra.run.smithi204.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-06T15:16:40.300 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2023-12-06T15:16:40.300 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-06T15:16:40.300 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-06T15:16:40.300 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:16:40.300 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:16:40.300 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:16:40.300 INFO:teuthology.orchestra.run.smithi204.stdout:Install 3 Packages 2023-12-06T15:16:40.300 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:16:40.301 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 90 M 2023-12-06T15:16:40.301 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 96 M 2023-12-06T15:16:40.301 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:16:42.146 INFO:teuthology.orchestra.run.smithi204.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 6.1 MB/s | 10 MB 00:01 2023-12-06T15:16:43.362 INFO:teuthology.orchestra.run.smithi204.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 13 MB/s | 36 MB 00:02 2023-12-06T15:16:43.772 INFO:teuthology.orchestra.run.smithi204.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 13 MB/s | 43 MB 00:03 2023-12-06T15:16:43.772 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:16:43.772 INFO:teuthology.orchestra.run.smithi204.stdout:Total 26 MB/s | 90 MB 00:03 2023-12-06T15:16:44.452 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:16:44.867 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:16:44.867 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:16:46.096 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:16:46.120 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:16:48.289 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - Extras 61 kB/s | 18 kB 00:00 2023-12-06T15:16:48.881 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-12-06T15:16:48.886 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:16:49.232 INFO:teuthology.orchestra.run.smithi176.stdout:Copr repo for python3-asyncssh owned by ceph 17 kB/s | 3.5 kB 00:00 2023-12-06T15:16:49.483 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-06T15:16:50.300 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-12-06T15:16:53.072 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-06T15:16:53.483 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-06T15:16:54.133 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 393 kB/s | 24 kB 00:00 2023-12-06T15:16:56.921 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-06T15:16:57.223 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-06T15:16:57.830 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-06T15:16:57.830 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-06T15:16:58.126 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:16:58.126 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:16:58.126 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2023-12-06T15:16:58.127 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:16:58.127 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:16:58.127 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-06T15:16:58.127 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-06T15:16:58.127 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-06T15:16:58.127 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-06T15:16:58.127 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:16:58.127 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:16:58.127 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:16:58.127 INFO:teuthology.orchestra.run.smithi176.stdout:Install 3 Packages 2023-12-06T15:16:58.127 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:16:58.128 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 90 M 2023-12-06T15:16:58.128 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 96 M 2023-12-06T15:16:58.128 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:16:59.343 INFO:teuthology.orchestra.run.smithi176.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 8.6 MB/s | 10 MB 00:01 2023-12-06T15:17:00.418 INFO:teuthology.orchestra.run.smithi176.stdout:(2/3): kernel-core-4.18.0-526.el8.x86_64.rpm 19 MB/s | 43 MB 00:02 2023-12-06T15:17:00.951 INFO:teuthology.orchestra.run.smithi176.stdout:(3/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 13 MB/s | 36 MB 00:02 2023-12-06T15:17:00.951 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:17:00.952 INFO:teuthology.orchestra.run.smithi176.stdout:Total 32 MB/s | 90 MB 00:02 2023-12-06T15:17:01.625 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:17:02.053 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:17:02.054 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:17:03.293 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:17:03.317 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:17:06.110 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:17:06.670 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-06T15:17:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-06T15:17:10.603 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-06T15:17:14.031 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-06T15:17:14.325 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-06T15:17:19.347 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-06T15:17:19.371 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-06T15:17:19.734 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-06T15:17:19.734 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-06T15:17:19.734 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-06T15:17:20.558 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-06T15:17:20.558 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:17:20.558 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:17:20.558 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-06T15:17:20.559 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-06T15:17:20.559 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:17:20.559 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:17:20.893 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T15:17:21.111 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-06T15:17:21.112 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-06T15:17:21.112 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-06T15:17:21.112 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-06T15:17:21.112 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-06T15:17:21.112 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2023-12-06T15:17:21.112 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2023-12-06T15:17:21.112 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T15:17:21.321 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-06T15:17:21.322 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-06T15:17:21.322 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -qi grub2-tools 2023-12-06T15:17:21.355 INFO:teuthology.orchestra.run.smithi204.stdout:Name : grub2-tools 2023-12-06T15:17:21.355 INFO:teuthology.orchestra.run.smithi204.stdout:Epoch : 1 2023-12-06T15:17:21.355 INFO:teuthology.orchestra.run.smithi204.stdout:Version : 2.02 2023-12-06T15:17:21.355 INFO:teuthology.orchestra.run.smithi204.stdout:Release : 129.el8 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: x86_64 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Group : System Environment/Base 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Size : 9265586 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:License : GPLv3+ 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Relocations : (not relocatable) 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Packager : CentOS BuildSystem 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Vendor : CentOS 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Summary : Support tools for GRUB. 2023-12-06T15:17:21.356 INFO:teuthology.orchestra.run.smithi204.stdout:Description : 2023-12-06T15:17:21.357 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:17:21.357 INFO:teuthology.orchestra.run.smithi204.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-06T15:17:21.357 INFO:teuthology.orchestra.run.smithi204.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-06T15:17:21.357 INFO:teuthology.orchestra.run.smithi204.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-06T15:17:21.357 INFO:teuthology.orchestra.run.smithi204.stdout:hardware devices. 2023-12-06T15:17:21.357 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:17:21.357 INFO:teuthology.orchestra.run.smithi204.stdout:This subpackage provides tools for support of all platforms. 2023-12-06T15:17:21.358 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-06T15:17:21.358 INFO:teuthology.task.kernel:Updating grub on smithi204 to boot 4.18.0-526.el8.x86_64 2023-12-06T15:17:21.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-06T15:17:22.807 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2023-12-06T15:17:24.951 INFO:teuthology.orchestra.run.smithi204.stderr:done 2023-12-06T15:17:24.953 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2023-12-06T15:17:24.978 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.rXsOd7rglS 2023-12-06T15:17:24.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rXsOd7rglS 2023-12-06T15:17:25.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.rXsOd7rglS 2023-12-06T15:17:25.190 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.rXsOd7rglS is 6KB 2023-12-06T15:17:25.238 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.rXsOd7rglS 2023-12-06T15:17:25.256 DEBUG:teuthology.orchestra.run.smithi204:> sudo /bin/ls /boot/loader/entries || true 2023-12-06T15:17:25.338 INFO:teuthology.orchestra.run.smithi204.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-06T15:17:25.338 INFO:teuthology.orchestra.run.smithi204.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-06T15:17:25.338 INFO:teuthology.orchestra.run.smithi204.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-06T15:17:25.338 INFO:teuthology.orchestra.run.smithi204.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-06T15:17:25.339 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-06T15:17:25.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2023-12-06T15:17:36.155 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-06T15:17:36.180 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-06T15:17:36.521 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-06T15:17:36.521 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-06T15:17:36.521 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-06T15:17:37.292 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-06T15:17:37.292 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:17:37.292 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:17:37.293 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-06T15:17:37.293 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-06T15:17:37.293 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:17:37.293 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:17:37.977 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T15:17:38.191 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-06T15:17:38.192 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-06T15:17:38.192 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-06T15:17:38.192 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-06T15:17:38.192 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-06T15:17:38.193 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2023-12-06T15:17:38.193 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2023-12-06T15:17:38.193 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T15:17:38.403 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-06T15:17:38.403 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-06T15:17:38.404 DEBUG:teuthology.orchestra.run.smithi176:> sudo rpm -qi grub2-tools 2023-12-06T15:17:38.437 INFO:teuthology.orchestra.run.smithi176.stdout:Name : grub2-tools 2023-12-06T15:17:38.437 INFO:teuthology.orchestra.run.smithi176.stdout:Epoch : 1 2023-12-06T15:17:38.437 INFO:teuthology.orchestra.run.smithi176.stdout:Version : 2.02 2023-12-06T15:17:38.437 INFO:teuthology.orchestra.run.smithi176.stdout:Release : 129.el8 2023-12-06T15:17:38.437 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: x86_64 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Group : System Environment/Base 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Size : 9265586 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:License : GPLv3+ 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Relocations : (not relocatable) 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Packager : CentOS BuildSystem 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Vendor : CentOS 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Summary : Support tools for GRUB. 2023-12-06T15:17:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Description : 2023-12-06T15:17:38.439 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:17:38.439 INFO:teuthology.orchestra.run.smithi176.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-06T15:17:38.439 INFO:teuthology.orchestra.run.smithi176.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-06T15:17:38.439 INFO:teuthology.orchestra.run.smithi176.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-06T15:17:38.439 INFO:teuthology.orchestra.run.smithi176.stdout:hardware devices. 2023-12-06T15:17:38.439 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:17:38.439 INFO:teuthology.orchestra.run.smithi176.stdout:This subpackage provides tools for support of all platforms. 2023-12-06T15:17:38.440 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-06T15:17:38.440 INFO:teuthology.task.kernel:Updating grub on smithi176 to boot 4.18.0-526.el8.x86_64 2023-12-06T15:17:38.440 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-06T15:17:39.788 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2023-12-06T15:17:41.927 INFO:teuthology.orchestra.run.smithi176.stderr:done 2023-12-06T15:17:41.929 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2023-12-06T15:17:41.954 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.jHPNIauJAG 2023-12-06T15:17:41.955 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jHPNIauJAG 2023-12-06T15:17:42.048 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.jHPNIauJAG 2023-12-06T15:17:42.161 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.jHPNIauJAG is 6KB 2023-12-06T15:17:42.209 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.jHPNIauJAG 2023-12-06T15:17:42.226 DEBUG:teuthology.orchestra.run.smithi176:> sudo /bin/ls /boot/loader/entries || true 2023-12-06T15:17:42.291 INFO:teuthology.orchestra.run.smithi176.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-06T15:17:42.291 INFO:teuthology.orchestra.run.smithi176.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-06T15:17:42.291 INFO:teuthology.orchestra.run.smithi176.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-06T15:17:42.291 INFO:teuthology.orchestra.run.smithi176.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-06T15:17:42.292 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-06T15:17:42.433 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2023-12-06T15:17:55.486 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2023-12-06T15:17:55.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-12-06T15:17:55.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T15:18:12.450 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2023-12-06T15:18:12.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-12-06T15:18:12.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T15:18:29.987 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2023-12-06T15:18:38.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-12-06T15:18:38.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T15:18:57.378 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2023-12-06T15:19:09.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-12-06T15:19:09.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T15:19:09.700 DEBUG:teuthology.orchestra.run.smithi204:> true 2023-12-06T15:19:09.927 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-12-06T15:19:09.927 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "4.18.0-526.el8.x86_64"... 2023-12-06T15:19:09.927 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2023-12-06T15:19:09.985 INFO:teuthology.orchestra.run.smithi204.stdout:4.18.0-526.el8.x86_64 2023-12-06T15:19:09.986 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-06T15:19:09.986 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-06T15:19:09.986 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-06T15:19:10.987 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-12-06T15:19:10.987 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-06T15:19:11.051 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2023-12-06T15:19:11.064 DEBUG:teuthology.parallel:result is None 2023-12-06T15:19:12.567 DEBUG:teuthology.orchestra.remote:timed out 2023-12-06T15:19:21.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-12-06T15:19:21.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T15:19:21.978 DEBUG:teuthology.orchestra.run.smithi176:> true 2023-12-06T15:19:22.212 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-12-06T15:19:22.212 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "4.18.0-526.el8.x86_64"... 2023-12-06T15:19:22.212 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2023-12-06T15:19:22.228 INFO:teuthology.orchestra.run.smithi176.stdout:4.18.0-526.el8.x86_64 2023-12-06T15:19:22.229 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-06T15:19:22.229 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-06T15:19:22.229 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-06T15:19:23.230 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-12-06T15:19:23.230 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-06T15:19:23.314 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2023-12-06T15:19:23.328 DEBUG:teuthology.parallel:result is None 2023-12-06T15:19:23.328 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-06T15:19:23.336 INFO:teuthology.task.internal:Creating test directory... 2023-12-06T15:19:23.336 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-06T15:19:23.338 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-06T15:19:23.362 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-06T15:19:23.371 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-06T15:19:23.385 INFO:teuthology.task.internal:Creating archive directory... 2023-12-06T15:19:23.385 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-06T15:19:23.396 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-06T15:19:23.451 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-06T15:19:23.457 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-06T15:19:23.457 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-06T15:19:23.467 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-06T15:19:23.507 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T15:19:23.517 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T15:19:23.526 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T15:19:23.537 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T15:19:23.539 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-06T15:19:23.545 INFO:teuthology.task.internal:Configuring sudo... 2023-12-06T15:19:23.545 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-06T15:19:23.561 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-06T15:19:23.609 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-06T15:19:23.620 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-06T15:19:23.620 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-06T15:19:23.628 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-06T15:19:23.668 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T15:19:23.725 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T15:19:23.808 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T15:19:23.874 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T15:19:23.951 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:19:23.951 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-06T15:19:23.993 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T15:19:24.039 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T15:19:24.094 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T15:19:24.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T15:19:24.223 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:19:24.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-06T15:19:24.291 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2023-12-06T15:19:24.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2023-12-06T15:19:24.410 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T15:19:25.108 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T15:19:25.188 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-06T15:19:25.203 INFO:teuthology.task.internal:Starting timer... 2023-12-06T15:19:25.204 INFO:teuthology.run_tasks:Running task pcp... 2023-12-06T15:19:25.220 INFO:teuthology.run_tasks:Running task selinux... 2023-12-06T15:19:25.231 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-06T15:19:25.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2023-12-06T15:19:25.674 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2023-12-06T15:19:25.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2023-12-06T15:19:25.747 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2023-12-06T15:19:25.749 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-06T15:19:25.749 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2023-12-06T15:19:25.769 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2023-12-06T15:19:25.769 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2023-12-06T15:19:25.816 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2023-12-06T15:19:25.816 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi176.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2023-12-06T15:19:25.816 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T15:19:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:19:25.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T15:19:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:19:25.871 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-06T15:19:25.872 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2023-12-06T15:19:25.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2023-12-06T15:19:25.937 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-06T15:19:25.947 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-06T15:19:25.947 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-06T15:19:25.964 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-06T15:19:25.965 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi176.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-06T15:26:56.818 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi176.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2023-12-06T15:26:56.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-12-06T15:26:56.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T15:26:56.891 DEBUG:teuthology.orchestra.run.smithi176:> true 2023-12-06T15:26:56.962 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-12-06T15:26:56.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-12-06T15:26:56.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T15:26:57.030 DEBUG:teuthology.orchestra.run.smithi204:> true 2023-12-06T15:26:57.057 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2023-12-06T15:26:57.057 INFO:teuthology.run_tasks:Running task clock... 2023-12-06T15:26:57.068 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-06T15:26:57.068 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-06T15:26:57.068 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T15:26:57.071 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-06T15:26:57.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T15:26:57.102 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-06T15:26:57.117 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-06T15:26:57.131 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-06T15:26:57.146 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-06T15:26:57.182 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2023-12-06T15:26:57.193 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2023-12-06T15:26:57.207 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-06T15:26:57.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2023-12-06T15:26:57.220 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2023-12-06T15:26:57.221 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-06T15:26:57.234 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-06T15:26:57.247 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-06T15:26:57.305 INFO:teuthology.orchestra.run.smithi176.stderr:bash: ntpq: command not found 2023-12-06T15:26:57.309 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T15:26:57.309 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2023-12-06T15:26:57.309 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T15:26:57.309 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T15:26:57.309 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T15:26:57.309 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T15:26:57.312 INFO:teuthology.orchestra.run.smithi204.stderr:bash: ntpq: command not found 2023-12-06T15:26:57.316 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T15:26:57.316 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2023-12-06T15:26:57.316 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T15:26:57.316 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T15:26:57.316 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T15:26:57.316 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T15:26:57.318 INFO:teuthology.run_tasks:Running task install... 2023-12-06T15:26:57.326 DEBUG:teuthology.task.install:project ceph 2023-12-06T15:26:57.326 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}} 2023-12-06T15:26:57.326 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'} 2023-12-06T15:26:57.326 INFO:teuthology.task.install:Using flavor: default 2023-12-06T15:26:57.333 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-06T15:26:57.333 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-12-06T15:26:57.333 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b', 'tag': None, 'wait_for_package': False} 2023-12-06T15:26:57.334 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T15:26:57.335 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b', 'tag': None, 'wait_for_package': False} 2023-12-06T15:26:57.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T15:26:57.623 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-12-05-0753-pacific/d03fca4f35d43f61b4679936aef7bd2f01cced0b/centos/8/flavors/default/ 2023-12-06T15:26:57.624 INFO:teuthology.task.install.rpm:Package version is 16.2.14-335.gd03fca4f 2023-12-06T15:26:57.631 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-12-05-0753-pacific/d03fca4f35d43f61b4679936aef7bd2f01cced0b/centos/8/flavors/default/ 2023-12-06T15:26:57.631 INFO:teuthology.task.install.rpm:Package version is 16.2.14-335.gd03fca4f 2023-12-06T15:26:57.748 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-12-05-0753-pacific/d03fca4f35d43f61b4679936aef7bd2f01cced0b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-12-05-0753-pacific/d03fca4f35d43f61b4679936aef7bd2f01cced0b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-12-05-0753-pacific/d03fca4f35d43f61b4679936aef7bd2f01cced0b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-06T15:26:57.749 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:26:57.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-06T15:26:57.778 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-12-06T15:26:57.778 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d03fca4f35d43f61b4679936aef7bd2f01cced0b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-06T15:26:57.803 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-12-05-0753-pacific/d03fca4f35d43f61b4679936aef7bd2f01cced0b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-12-05-0753-pacific/d03fca4f35d43f61b4679936aef7bd2f01cced0b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-12-05-0753-pacific/d03fca4f35d43f61b4679936aef7bd2f01cced0b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-06T15:26:57.803 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:26:57.804 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-06T15:26:57.834 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-12-06T15:26:57.834 DEBUG:teuthology.orchestra.run.smithi176:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d03fca4f35d43f61b4679936aef7bd2f01cced0b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-06T15:26:57.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-06T15:26:57.905 DEBUG:teuthology.orchestra.run.smithi176:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-06T15:26:57.924 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-06T15:26:57.985 DEBUG:teuthology.orchestra.run.smithi176:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-06T15:26:57.991 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2023-12-06T15:26:57.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2023-12-06T15:26:58.054 INFO:teuthology.orchestra.run.smithi176.stdout:check_obsoletes = 1 2023-12-06T15:26:58.055 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum clean all 2023-12-06T15:26:58.255 INFO:teuthology.orchestra.run.smithi204.stdout:76 files removed 2023-12-06T15:26:58.283 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw 2023-12-06T15:26:58.324 INFO:teuthology.orchestra.run.smithi176.stdout:76 files removed 2023-12-06T15:26:58.352 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw 2023-12-06T15:26:59.048 INFO:teuthology.orchestra.run.smithi176.stdout:ceph packages for x86_64 269 kB/s | 76 kB 00:00 2023-12-06T15:26:59.049 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 211 kB/s | 76 kB 00:00 2023-12-06T15:26:59.416 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 47 kB/s | 15 kB 00:00 2023-12-06T15:26:59.440 INFO:teuthology.orchestra.run.smithi176.stdout:ceph noarch packages 43 kB/s | 15 kB 00:00 2023-12-06T15:26:59.749 INFO:teuthology.orchestra.run.smithi176.stdout:ceph source packages 5.1 kB/s | 1.4 kB 00:00 2023-12-06T15:26:59.751 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 4.7 kB/s | 1.4 kB 00:00 2023-12-06T15:27:01.926 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - AppStream 16 MB/s | 34 MB 00:02 2023-12-06T15:27:01.975 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - AppStream 16 MB/s | 34 MB 00:02 2023-12-06T15:27:10.552 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 55 MB 00:02 2023-12-06T15:27:11.975 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 55 MB 00:03 2023-12-06T15:27:18.677 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - Extras 68 kB/s | 18 kB 00:00 2023-12-06T15:27:19.570 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - Extras common packages 9.7 kB/s | 6.9 kB 00:00 2023-12-06T15:27:20.183 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - Extras 46 kB/s | 18 kB 00:00 2023-12-06T15:27:20.859 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-12-06T15:27:21.137 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 8 - PowerTools 4.7 MB/s | 6.6 MB 00:01 2023-12-06T15:27:22.631 INFO:teuthology.orchestra.run.smithi204.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-12-06T15:27:25.159 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 6.7 MB/s | 16 MB 00:02 2023-12-06T15:27:26.143 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - PowerTools 1.3 MB/s | 6.6 MB 00:05 2023-12-06T15:27:27.661 INFO:teuthology.orchestra.run.smithi176.stdout:Copr repo for python3-asyncssh owned by ceph 17 kB/s | 3.5 kB 00:00 2023-12-06T15:27:29.048 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 416 kB/s | 24 kB 00:00 2023-12-06T15:27:33.630 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:27:33.633 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:27:33.633 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2023-12-06T15:27:33.633 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:27:33.633 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:27:33.633 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 12 M 2023-12-06T15:27:33.633 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2023-12-06T15:27:33.633 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 3.7 M 2023-12-06T15:27:33.633 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 4.2 M 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 6.1 M 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 24 M 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 26 k 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 824 k 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 493 k 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 3.8 M 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-06T15:27:33.634 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 45 k 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 99 k 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 215 k 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 389 k 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 369 k 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 114 k 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout:Install 21 Packages 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 2 Packages 2023-12-06T15:27:33.635 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:27:33.636 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 57 M 2023-12-06T15:27:33.636 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:27:33.989 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 2.6 MB/s | 16 MB 00:06 2023-12-06T15:27:36.874 INFO:teuthology.orchestra.run.smithi204.stdout:(1/23): ceph-base-16.2.14-335.gd03fca4f.el8.x86 2.1 MB/s | 6.1 MB 00:02 2023-12-06T15:27:36.925 INFO:teuthology.orchestra.run.smithi204.stdout:(2/23): ceph-selinux-16.2.14-335.gd03fca4f.el8. 515 kB/s | 26 kB 00:00 2023-12-06T15:27:37.292 INFO:teuthology.orchestra.run.smithi204.stdout:(3/23): libcephfs2-16.2.14-335.gd03fca4f.el8.x8 2.2 MB/s | 824 kB 00:00 2023-12-06T15:27:37.542 INFO:teuthology.orchestra.run.smithi204.stdout:(4/23): libradosstriper1-16.2.14-335.gd03fca4f. 1.9 MB/s | 493 kB 00:00 2023-12-06T15:27:37.853 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 390 kB/s | 24 kB 00:00 2023-12-06T15:27:38.526 INFO:teuthology.orchestra.run.smithi204.stdout:(5/23): librgw2-16.2.14-335.gd03fca4f.el8.x86_6 3.9 MB/s | 3.8 MB 00:00 2023-12-06T15:27:38.575 INFO:teuthology.orchestra.run.smithi204.stdout:(6/23): python3-ceph-argparse-16.2.14-335.gd03f 907 kB/s | 45 kB 00:00 2023-12-06T15:27:38.643 INFO:teuthology.orchestra.run.smithi204.stdout:(7/23): python3-ceph-common-16.2.14-335.gd03fca 1.4 MB/s | 99 kB 00:00 2023-12-06T15:27:38.743 INFO:teuthology.orchestra.run.smithi204.stdout:(8/23): python3-cephfs-16.2.14-335.gd03fca4f.el 2.1 MB/s | 215 kB 00:00 2023-12-06T15:27:38.877 INFO:teuthology.orchestra.run.smithi204.stdout:(9/23): python3-rados-16.2.14-335.gd03fca4f.el8 2.9 MB/s | 389 kB 00:00 2023-12-06T15:27:39.535 INFO:teuthology.orchestra.run.smithi204.stdout:(10/23): ceph-common-16.2.14-335.gd03fca4f.el8. 4.4 MB/s | 24 MB 00:05 2023-12-06T15:27:39.560 INFO:teuthology.orchestra.run.smithi204.stdout:(11/23): python3-rbd-16.2.14-335.gd03fca4f.el8. 540 kB/s | 369 kB 00:00 2023-12-06T15:27:39.619 INFO:teuthology.orchestra.run.smithi204.stdout:(12/23): python3-rgw-16.2.14-335.gd03fca4f.el8. 1.3 MB/s | 114 kB 00:00 2023-12-06T15:27:39.986 INFO:teuthology.orchestra.run.smithi204.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 831 kB/s | 354 kB 00:00 2023-12-06T15:27:40.020 INFO:teuthology.orchestra.run.smithi204.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 648 kB/s | 259 kB 00:00 2023-12-06T15:27:40.070 INFO:teuthology.orchestra.run.smithi204.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 525 kB/s | 44 kB 00:00 2023-12-06T15:27:40.462 INFO:teuthology.orchestra.run.smithi204.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 106 kB/s | 47 kB 00:00 2023-12-06T15:27:40.954 INFO:teuthology.orchestra.run.smithi204.stdout:(17/23): ceph-radosgw-16.2.14-335.gd03fca4f.el8 1.7 MB/s | 12 MB 00:06 2023-12-06T15:27:41.771 INFO:teuthology.orchestra.run.smithi204.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 180 kB/s | 306 kB 00:01 2023-12-06T15:27:42.063 INFO:teuthology.orchestra.run.smithi204.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 113 kB/s | 181 kB 00:01 2023-12-06T15:27:42.089 INFO:teuthology.orchestra.run.smithi204.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 237 kB/s | 75 kB 00:00 2023-12-06T15:27:42.172 INFO:teuthology.orchestra.run.smithi204.stdout:(21/23): liboath-2.6.2-3.el8.x86_64.rpm 48 kB/s | 59 kB 00:01 2023-12-06T15:27:42.445 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:27:42.447 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 12 M 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 3.7 M 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1 x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 4.2 M 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 6.1 M 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 24 M 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 26 k 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2 x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 824 k 2023-12-06T15:27:42.448 INFO:teuthology.orchestra.run.smithi176.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 493 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 3.8 M 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 45 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 99 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 215 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 389 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 369 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 114 k 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:27:42.449 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:27:42.450 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:27:42.450 INFO:teuthology.orchestra.run.smithi176.stdout:Install 21 Packages 2023-12-06T15:27:42.450 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 2 Packages 2023-12-06T15:27:42.450 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:27:42.450 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 57 M 2023-12-06T15:27:42.450 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:27:42.698 INFO:teuthology.orchestra.run.smithi204.stdout:(22/23): librados2-16.2.14-335.gd03fca4f.el8.x8 5.9 MB/s | 3.7 MB 00:00 2023-12-06T15:27:42.756 INFO:teuthology.orchestra.run.smithi204.stdout:(23/23): librbd1-16.2.14-335.gd03fca4f.el8.x86_ 6.3 MB/s | 4.2 MB 00:00 2023-12-06T15:27:42.760 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:27:42.761 INFO:teuthology.orchestra.run.smithi204.stdout:Total 6.3 MB/s | 57 MB 00:09 2023-12-06T15:27:42.803 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:27:42.851 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:27:42.851 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:27:43.203 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:27:43.206 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:27:43.787 INFO:teuthology.orchestra.run.smithi176.stdout:(1/23): ceph-radosgw-16.2.14-335.gd03fca4f.el8. 16 MB/s | 12 MB 00:00 2023-12-06T15:27:43.831 INFO:teuthology.orchestra.run.smithi176.stdout:(2/23): ceph-selinux-16.2.14-335.gd03fca4f.el8. 620 kB/s | 26 kB 00:00 2023-12-06T15:27:43.905 INFO:teuthology.orchestra.run.smithi176.stdout:(3/23): libcephfs2-16.2.14-335.gd03fca4f.el8.x8 11 MB/s | 824 kB 00:00 2023-12-06T15:27:43.955 INFO:teuthology.orchestra.run.smithi176.stdout:(4/23): libradosstriper1-16.2.14-335.gd03fca4f. 9.6 MB/s | 493 kB 00:00 2023-12-06T15:27:44.164 INFO:teuthology.orchestra.run.smithi176.stdout:(5/23): librgw2-16.2.14-335.gd03fca4f.el8.x86_6 18 MB/s | 3.8 MB 00:00 2023-12-06T15:27:44.240 INFO:teuthology.orchestra.run.smithi176.stdout:(6/23): python3-ceph-argparse-16.2.14-335.gd03f 594 kB/s | 45 kB 00:00 2023-12-06T15:27:44.286 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:27:44.289 INFO:teuthology.orchestra.run.smithi176.stdout:(7/23): python3-ceph-common-16.2.14-335.gd03fca 2.0 MB/s | 99 kB 00:00 2023-12-06T15:27:44.340 INFO:teuthology.orchestra.run.smithi176.stdout:(8/23): python3-cephfs-16.2.14-335.gd03fca4f.el 4.2 MB/s | 215 kB 00:00 2023-12-06T15:27:44.390 INFO:teuthology.orchestra.run.smithi176.stdout:(9/23): python3-rados-16.2.14-335.gd03fca4f.el8 7.6 MB/s | 389 kB 00:00 2023-12-06T15:27:44.440 INFO:teuthology.orchestra.run.smithi176.stdout:(10/23): python3-rbd-16.2.14-335.gd03fca4f.el8. 7.2 MB/s | 369 kB 00:00 2023-12-06T15:27:44.482 INFO:teuthology.orchestra.run.smithi176.stdout:(11/23): python3-rgw-16.2.14-335.gd03fca4f.el8. 2.7 MB/s | 114 kB 00:00 2023-12-06T15:27:44.503 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-12-06T15:27:44.722 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-06T15:27:45.104 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-06T15:27:45.255 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:16.2.14-335.gd03fca4f.el8.x86_64 3/25 2023-12-06T15:27:45.388 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.14-335.gd03fca4f.el8.x86_64 3/25 2023-12-06T15:27:45.628 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:16.2.14-335.gd03fca4f.el8.x86_64 4/25 2023-12-06T15:27:45.780 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:16.2.14-335.gd03fca4f.el8.x86_64 5/25 2023-12-06T15:27:45.991 INFO:teuthology.orchestra.run.smithi176.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 235 kB/s | 354 kB 00:01 2023-12-06T15:27:46.189 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:16.2.14-335.gd03fca4f.el8.x86_64 5/25 2023-12-06T15:27:46.389 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:16.2.14-335.gd03fca4f.el8.x86_64 6/25 2023-12-06T15:27:46.453 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.14-335.gd03fca4f.el8.x86_64 6/25 2023-12-06T15:27:46.607 INFO:teuthology.orchestra.run.smithi176.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 421 kB/s | 259 kB 00:00 2023-12-06T15:27:46.624 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-12-06T15:27:46.759 INFO:teuthology.orchestra.run.smithi176.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 290 kB/s | 44 kB 00:00 2023-12-06T15:27:46.788 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-12-06T15:27:47.150 INFO:teuthology.orchestra.run.smithi176.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 120 kB/s | 47 kB 00:00 2023-12-06T15:27:47.173 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-12-06T15:27:47.322 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:16.2.14-335.gd03fca4f.el8.x86_64 9/25 2023-12-06T15:27:47.411 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:16.2.14-335.gd03fca4f.el8.x86_64 9/25 2023-12-06T15:27:47.586 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-12-06T15:27:47.667 INFO:teuthology.orchestra.run.smithi176.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 593 kB/s | 306 kB 00:00 2023-12-06T15:27:47.735 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:16.2.14-335.gd03fca4f.el8. 11/25 2023-12-06T15:27:47.884 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:16.2.14-335.gd03fca4f.el8.x86_64 12/25 2023-12-06T15:27:47.967 INFO:teuthology.orchestra.run.smithi176.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 603 kB/s | 181 kB 00:00 2023-12-06T15:27:48.084 INFO:teuthology.orchestra.run.smithi176.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 505 kB/s | 59 kB 00:00 2023-12-06T15:27:48.098 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:16.2.14-335.gd03fca4f.el8.x86_64 13/25 2023-12-06T15:27:48.226 INFO:teuthology.orchestra.run.smithi176.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 530 kB/s | 75 kB 00:00 2023-12-06T15:27:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:16.2.14-335.gd03fca4f.el8.x86_64 14/25 2023-12-06T15:27:48.430 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:16.2.14-335.gd03fca4f.el8.x86_6 15/25 2023-12-06T15:27:48.490 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:16.2.14-335.gd03fca4f.el8.x86_6 15/25 2023-12-06T15:27:48.526 INFO:teuthology.orchestra.run.smithi176.stdout:(20/23): librados2-16.2.14-335.gd03fca4f.el8.x8 12 MB/s | 3.7 MB 00:00 2023-12-06T15:27:48.689 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-12-06T15:27:48.743 INFO:teuthology.orchestra.run.smithi176.stdout:(21/23): librbd1-16.2.14-335.gd03fca4f.el8.x86_ 19 MB/s | 4.2 MB 00:00 2023-12-06T15:27:48.863 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-12-06T15:27:49.057 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-12-06T15:27:49.214 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:16.2.14-335.gd03fca4f.el8.x8 19/25 2023-12-06T15:27:50.543 INFO:teuthology.orchestra.run.smithi176.stdout:(22/23): ceph-base-16.2.14-335.gd03fca4f.el8.x8 829 kB/s | 6.1 MB 00:07 2023-12-06T15:27:51.729 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:16.2.14-335.gd03fca4f.el8.x86_64 20/25 2023-12-06T15:27:51.930 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:16.2.14-335.gd03fca4f.el8.x86_64 20/25 2023-12-06T15:27:52.621 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:16.2.14-335.gd03fca4f.el8.x86_64 20/25 2023-12-06T15:27:52.875 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:16.2.14-335.gd03fca4f.el8.x86_64 21/25 2023-12-06T15:27:52.981 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:16.2.14-335.gd03fca4f.el8.x86_64 21/25 2023-12-06T15:27:53.104 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:16.2.14-335.gd03fca4f.el8.x86_64 22/25 2023-12-06T15:27:55.334 INFO:teuthology.orchestra.run.smithi176.stdout:(23/23): ceph-common-16.2.14-335.gd03fca4f.el8. 2.0 MB/s | 24 MB 00:12 2023-12-06T15:27:55.337 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:27:55.337 INFO:teuthology.orchestra.run.smithi176.stdout:Total 4.5 MB/s | 57 MB 00:12 2023-12-06T15:27:55.380 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:27:55.427 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:27:55.428 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:27:55.780 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:27:55.784 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:27:56.912 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:27:57.137 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-12-06T15:27:57.339 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-06T15:27:57.716 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-06T15:27:57.880 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librados2-2:16.2.14-335.gd03fca4f.el8.x86_64 3/25 2023-12-06T15:27:57.981 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:16.2.14-335.gd03fca4f.el8.x86_64 3/25 2023-12-06T15:27:58.261 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-2:16.2.14-335.gd03fca4f.el8.x86_64 4/25 2023-12-06T15:27:58.464 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-2:16.2.14-335.gd03fca4f.el8.x86_64 5/25 2023-12-06T15:27:58.873 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-2:16.2.14-335.gd03fca4f.el8.x86_64 5/25 2023-12-06T15:27:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librbd1-2:16.2.14-335.gd03fca4f.el8.x86_64 6/25 2023-12-06T15:27:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:16.2.14-335.gd03fca4f.el8.x86_64 6/25 2023-12-06T15:27:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-12-06T15:27:59.563 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-12-06T15:27:59.949 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-12-06T15:28:00.130 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librgw2-2:16.2.14-335.gd03fca4f.el8.x86_64 9/25 2023-12-06T15:28:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librgw2-2:16.2.14-335.gd03fca4f.el8.x86_64 9/25 2023-12-06T15:28:00.418 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-12-06T15:28:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-argparse-2:16.2.14-335.gd03fca4f.el8. 11/25 2023-12-06T15:28:00.783 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-2:16.2.14-335.gd03fca4f.el8.x86_64 12/25 2023-12-06T15:28:01.039 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-2:16.2.14-335.gd03fca4f.el8.x86_64 13/25 2023-12-06T15:28:01.262 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-2:16.2.14-335.gd03fca4f.el8.x86_64 14/25 2023-12-06T15:28:01.438 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libradosstriper1-2:16.2.14-335.gd03fca4f.el8.x86_6 15/25 2023-12-06T15:28:01.509 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libradosstriper1-2:16.2.14-335.gd03fca4f.el8.x86_6 15/25 2023-12-06T15:28:01.713 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-12-06T15:28:01.963 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-12-06T15:28:02.190 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-12-06T15:28:02.380 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-common-2:16.2.14-335.gd03fca4f.el8.x8 19/25 2023-12-06T15:28:04.866 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:16.2.14-335.gd03fca4f.el8.x86_64 20/25 2023-12-06T15:28:05.105 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-2:16.2.14-335.gd03fca4f.el8.x86_64 20/25 2023-12-06T15:28:05.799 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:16.2.14-335.gd03fca4f.el8.x86_64 20/25 2023-12-06T15:28:06.111 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-2:16.2.14-335.gd03fca4f.el8.x86_64 21/25 2023-12-06T15:28:06.213 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-2:16.2.14-335.gd03fca4f.el8.x86_64 21/25 2023-12-06T15:28:06.354 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-selinux-2:16.2.14-335.gd03fca4f.el8.x86_64 22/25 2023-12-06T15:28:11.377 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:16.2.14-335.gd03fca4f.el8.x86_64 22/25 2023-12-06T15:28:11.377 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2023-12-06T15:28:11.377 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2023-12-06T15:28:11.377 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /dev 2023-12-06T15:28:11.377 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /run 2023-12-06T15:28:11.377 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2023-12-06T15:28:11.377 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2023-12-06T15:28:11.377 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2023-12-06T15:28:11.377 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2023-12-06T15:28:11.377 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /dev 2023-12-06T15:28:11.377 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:12.531 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:16.2.14-335.gd03fca4f.el8.x86_64 23/25 2023-12-06T15:28:12.615 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-335.gd03fca4f.el8.x86_64 23/25 2023-12-06T15:28:12.615 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-06T15:28:12.812 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-06T15:28:12.813 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-06T15:28:13.493 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-06T15:28:13.493 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:16.2.14-335.gd03fca4f.el8.x86_64 1/25 2023-12-06T15:28:13.493 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:16.2.14-335.gd03fca4f.el8.x86_64 2/25 2023-12-06T15:28:13.493 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:16.2.14-335.gd03fca4f.el8.x86_64 3/25 2023-12-06T15:28:13.493 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:16.2.14-335.gd03fca4f.el8.x86_64 4/25 2023-12-06T15:28:13.493 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:16.2.14-335.gd03fca4f.el8.x86_64 5/25 2023-12-06T15:28:13.493 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:16.2.14-335.gd03fca4f.el8.x86_6 6/25 2023-12-06T15:28:13.493 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:16.2.14-335.gd03fca4f.el8.x86_64 7/25 2023-12-06T15:28:13.493 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:16.2.14-335.gd03fca4f.el8. 8/25 2023-12-06T15:28:13.493 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:16.2.14-335.gd03fca4f.el8.x8 9/25 2023-12-06T15:28:13.493 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:16.2.14-335.gd03fca4f.el8.x86_64 10/25 2023-12-06T15:28:13.493 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:16.2.14-335.gd03fca4f.el8.x86_64 11/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:16.2.14-335.gd03fca4f.el8.x86_64 12/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:16.2.14-335.gd03fca4f.el8.x86_64 13/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.14-335.gd03fca4f.el8.x86_64 22/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-12-06T15:28:13.494 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.14-335.gd03fca4f.el8.x86_64 24/25 2023-12-06T15:28:14.586 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-12-06T15:28:14.586 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:14.586 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: leveldb-1.22-1.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-12-06T15:28:14.587 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.588 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-06T15:28:14.588 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-06T15:28:14.588 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.588 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.588 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.588 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-06T15:28:14.588 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.588 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.588 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:14.588 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:14.588 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:28:14.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-test 2023-12-06T15:28:15.633 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:47 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:28:17.325 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:28:17.326 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:28:17.326 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2023-12-06T15:28:17.326 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:28:17.326 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:28:17.326 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 52 M 2023-12-06T15:28:17.326 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2023-12-06T15:28:17.327 INFO:teuthology.orchestra.run.smithi204.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-06T15:28:17.327 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 174 k 2023-12-06T15:28:17.327 INFO:teuthology.orchestra.run.smithi204.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-06T15:28:17.327 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-06T15:28:17.327 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-06T15:28:17.327 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:17.327 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:28:17.327 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:28:17.327 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2023-12-06T15:28:17.327 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:17.328 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 53 M 2023-12-06T15:28:17.328 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 196 M 2023-12-06T15:28:17.328 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:28:17.732 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): libcephsqlite-16.2.14-335.gd03fca4f.el8. 796 kB/s | 174 kB 00:00 2023-12-06T15:28:17.765 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 808 kB/s | 203 kB 00:00 2023-12-06T15:28:17.899 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2023-12-06T15:28:17.941 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 899 kB/s | 187 kB 00:00 2023-12-06T15:28:17.958 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-12-06T15:28:20.300 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): ceph-test-16.2.14-335.gd03fca4f.el8.x86_ 19 MB/s | 52 MB 00:02 2023-12-06T15:28:20.301 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:28:20.302 INFO:teuthology.orchestra.run.smithi204.stdout:Total 18 MB/s | 53 MB 00:02 2023-12-06T15:28:20.345 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:28:20.366 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:28:20.366 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:28:20.708 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:28:20.710 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:28:21.180 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:28:21.420 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-06T15:28:21.586 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-06T15:28:21.684 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-06T15:28:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-06T15:28:21.941 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-06T15:28:22.084 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:16.2.14-335.gd03fca4f.el8.x86_64 5/6 2023-12-06T15:28:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-selinux-2:16.2.14-335.gd03fca4f.el8.x86_64 22/25 2023-12-06T15:28:24.396 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /sys 2023-12-06T15:28:24.396 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /proc 2023-12-06T15:28:24.396 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev 2023-12-06T15:28:24.396 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run 2023-12-06T15:28:24.396 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2023-12-06T15:28:24.396 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /var/tmp 2023-12-06T15:28:24.396 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /home 2023-12-06T15:28:24.396 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /tmp 2023-12-06T15:28:24.397 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev 2023-12-06T15:28:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:28:25.566 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-2:16.2.14-335.gd03fca4f.el8.x86_64 23/25 2023-12-06T15:28:25.648 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-335.gd03fca4f.el8.x86_64 23/25 2023-12-06T15:28:25.648 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-06T15:28:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-06T15:28:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-06T15:28:26.657 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-06T15:28:26.657 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-2:16.2.14-335.gd03fca4f.el8.x86_64 1/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-2:16.2.14-335.gd03fca4f.el8.x86_64 2/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-2:16.2.14-335.gd03fca4f.el8.x86_64 3/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-selinux-2:16.2.14-335.gd03fca4f.el8.x86_64 4/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-2:16.2.14-335.gd03fca4f.el8.x86_64 5/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libradosstriper1-2:16.2.14-335.gd03fca4f.el8.x86_6 6/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librgw2-2:16.2.14-335.gd03fca4f.el8.x86_64 7/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-argparse-2:16.2.14-335.gd03fca4f.el8. 8/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-common-2:16.2.14-335.gd03fca4f.el8.x8 9/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-2:16.2.14-335.gd03fca4f.el8.x86_64 10/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-2:16.2.14-335.gd03fca4f.el8.x86_64 11/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-2:16.2.14-335.gd03fca4f.el8.x86_64 12/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-2:16.2.14-335.gd03fca4f.el8.x86_64 13/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-12-06T15:28:26.658 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-12-06T15:28:26.659 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-12-06T15:28:26.659 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-12-06T15:28:26.659 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-12-06T15:28:26.659 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2023-12-06T15:28:26.659 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2023-12-06T15:28:26.659 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2023-12-06T15:28:26.659 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:16.2.14-335.gd03fca4f.el8.x86_64 22/25 2023-12-06T15:28:26.659 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-12-06T15:28:26.659 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:16.2.14-335.gd03fca4f.el8.x86_64 24/25 2023-12-06T15:28:26.683 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:16.2.14-335.gd03fca4f.el8.x86_64 5/6 2023-12-06T15:28:26.924 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:16.2.14-335.gd03fca4f.el8.x86_64 6/6 2023-12-06T15:28:27.210 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-test-2:16.2.14-335.gd03fca4f.el8.x86_64 6/6 2023-12-06T15:28:27.210 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:16.2.14-335.gd03fca4f.el8.x86_64 1/6 2023-12-06T15:28:27.210 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:16.2.14-335.gd03fca4f.el8.x86_64 2/6 2023-12-06T15:28:27.210 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-06T15:28:27.210 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-06T15:28:27.210 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-06T15:28:27.821 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-12-06T15:28:27.821 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:28:27.821 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2023-12-06T15:28:27.821 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: leveldb-1.22-1.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-12-06T15:28:27.822 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-06T15:28:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-06T15:28:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-06T15:28:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.823 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:28:27.823 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:28:27.993 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-06T15:28:27.994 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:27.994 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:28:27.994 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.994 INFO:teuthology.orchestra.run.smithi204.stdout: jq-1.6-8.el8.x86_64 2023-12-06T15:28:27.994 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:27.994 INFO:teuthology.orchestra.run.smithi204.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-12-06T15:28:27.994 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-1.el8.x86_64 2023-12-06T15:28:27.994 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-12-06T15:28:27.994 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:27.994 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:28:28.068 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-test 2023-12-06T15:28:28.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph 2023-12-06T15:28:28.911 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:51 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:28:29.032 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:01 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:28:30.576 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:28:30.577 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:28:30.577 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-06T15:28:30.577 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:28:30.577 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:28:30.577 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 52 M 2023-12-06T15:28:30.577 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-06T15:28:30.577 INFO:teuthology.orchestra.run.smithi176.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-06T15:28:30.577 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 174 k 2023-12-06T15:28:30.577 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-06T15:28:30.577 INFO:teuthology.orchestra.run.smithi176.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-06T15:28:30.578 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-06T15:28:30.578 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:28:30.578 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:28:30.578 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:28:30.578 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2023-12-06T15:28:30.578 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:28:30.578 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 53 M 2023-12-06T15:28:30.578 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 196 M 2023-12-06T15:28:30.579 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:28:30.706 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:28:30.710 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:28:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2023-12-06T15:28:30.710 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:28:30.710 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:28:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 6.4 k 2023-12-06T15:28:30.710 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2023-12-06T15:28:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 2.4 M 2023-12-06T15:28:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 1.6 M 2023-12-06T15:28:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 245 k 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 5.0 M 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 18 M 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-06T15:28:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-06T15:28:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:30.713 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:28:30.713 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:28:30.713 INFO:teuthology.orchestra.run.smithi204.stdout:Install 32 Packages 2023-12-06T15:28:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:30.713 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 31 M 2023-12-06T15:28:30.713 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 101 M 2023-12-06T15:28:30.713 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:28:30.831 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): libcephsqlite-16.2.14-335.gd03fca4f.el8. 820 kB/s | 174 kB 00:00 2023-12-06T15:28:30.872 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 804 kB/s | 203 kB 00:00 2023-12-06T15:28:31.005 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2023-12-06T15:28:31.047 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 868 kB/s | 187 kB 00:00 2023-12-06T15:28:31.072 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2023-12-06T15:28:31.394 INFO:teuthology.orchestra.run.smithi204.stdout:(1/32): ceph-16.2.14-335.gd03fca4f.el8.x86_64.r 23 kB/s | 6.4 kB 00:00 2023-12-06T15:28:32.186 INFO:teuthology.orchestra.run.smithi204.stdout:(2/32): ceph-mds-16.2.14-335.gd03fca4f.el8.x86_ 2.3 MB/s | 2.4 MB 00:01 2023-12-06T15:28:33.127 INFO:teuthology.orchestra.run.smithi204.stdout:(3/32): ceph-mgr-16.2.14-335.gd03fca4f.el8.x86_ 827 kB/s | 1.6 MB 00:02 2023-12-06T15:28:33.253 INFO:teuthology.orchestra.run.smithi204.stdout:(4/32): ceph-mgr-modules-core-16.2.14-335.gd03f 1.9 MB/s | 245 kB 00:00 2023-12-06T15:28:33.454 INFO:teuthology.orchestra.run.smithi204.stdout:(5/32): ceph-mon-16.2.14-335.gd03fca4f.el8.x86_ 2.4 MB/s | 5.0 MB 00:02 2023-12-06T15:28:33.523 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): ceph-test-16.2.14-335.gd03fca4f.el8.x86_ 18 MB/s | 52 MB 00:02 2023-12-06T15:28:33.524 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:28:33.525 INFO:teuthology.orchestra.run.smithi176.stdout:Total 18 MB/s | 53 MB 00:02 2023-12-06T15:28:33.569 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:28:33.589 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:28:33.589 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:28:33.695 INFO:teuthology.orchestra.run.smithi204.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 161 kB/s | 39 kB 00:00 2023-12-06T15:28:33.720 INFO:teuthology.orchestra.run.smithi204.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 336 kB/s | 157 kB 00:00 2023-12-06T15:28:33.854 INFO:teuthology.orchestra.run.smithi204.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 653 kB/s | 103 kB 00:00 2023-12-06T15:28:33.879 INFO:teuthology.orchestra.run.smithi204.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 360 kB/s | 57 kB 00:00 2023-12-06T15:28:33.935 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:28:33.938 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:28:34.055 INFO:teuthology.orchestra.run.smithi204.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.2 MB/s | 457 kB 00:00 2023-12-06T15:28:34.388 INFO:teuthology.orchestra.run.smithi204.stdout:(11/32): ceph-osd-16.2.14-335.gd03fca4f.el8.x86 8.3 MB/s | 18 MB 00:02 2023-12-06T15:28:34.407 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:28:34.413 INFO:teuthology.orchestra.run.smithi204.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 466 kB/s | 249 kB 00:00 2023-12-06T15:28:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-06T15:28:34.780 INFO:teuthology.orchestra.run.smithi204.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 119 kB/s | 44 kB 00:00 2023-12-06T15:28:34.881 INFO:teuthology.orchestra.run.smithi204.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 222 kB/s | 109 kB 00:00 2023-12-06T15:28:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-06T15:28:34.922 INFO:teuthology.orchestra.run.smithi204.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.3 MB/s | 185 kB 00:00 2023-12-06T15:28:35.041 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-06T15:28:35.120 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-06T15:28:35.156 INFO:teuthology.orchestra.run.smithi204.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2023-12-06T15:28:35.190 INFO:teuthology.orchestra.run.smithi204.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 558 kB/s | 173 kB 00:00 2023-12-06T15:28:35.207 INFO:teuthology.orchestra.run.smithi204.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 220 kB/s | 11 kB 00:00 2023-12-06T15:28:35.249 INFO:teuthology.orchestra.run.smithi204.stdout:(19/32): python3-cryptography-3.2.1-6.el8.x86_6 537 kB/s | 641 kB 00:01 2023-12-06T15:28:35.274 INFO:teuthology.orchestra.run.smithi204.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 209 kB/s | 18 kB 00:00 2023-12-06T15:28:35.299 INFO:teuthology.orchestra.run.smithi204.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 528 kB/s | 49 kB 00:00 2023-12-06T15:28:35.352 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-06T15:28:35.358 INFO:teuthology.orchestra.run.smithi204.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 290 kB/s | 16 kB 00:00 2023-12-06T15:28:35.417 INFO:teuthology.orchestra.run.smithi204.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 325 kB/s | 19 kB 00:00 2023-12-06T15:28:35.442 INFO:teuthology.orchestra.run.smithi204.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2023-12-06T15:28:35.477 INFO:teuthology.orchestra.run.smithi204.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 408 kB/s | 24 kB 00:00 2023-12-06T15:28:35.494 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephsqlite-2:16.2.14-335.gd03fca4f.el8.x86_64 5/6 2023-12-06T15:28:35.501 INFO:teuthology.orchestra.run.smithi204.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 499 kB/s | 29 kB 00:00 2023-12-06T15:28:35.535 INFO:teuthology.orchestra.run.smithi204.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 470 kB/s | 27 kB 00:00 2023-12-06T15:28:35.560 INFO:teuthology.orchestra.run.smithi204.stdout:(28/32): python3-more-itertools-7.2.0-3.el8.noa 189 kB/s | 59 kB 00:00 2023-12-06T15:28:35.660 INFO:teuthology.orchestra.run.smithi204.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 854 kB/s | 85 kB 00:00 2023-12-06T15:28:35.685 INFO:teuthology.orchestra.run.smithi204.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.4 MB/s | 255 kB 00:00 2023-12-06T15:28:35.719 INFO:teuthology.orchestra.run.smithi204.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2023-12-06T15:28:35.744 INFO:teuthology.orchestra.run.smithi204.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 274 kB/s | 23 kB 00:00 2023-12-06T15:28:35.748 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:28:35.748 INFO:teuthology.orchestra.run.smithi204.stdout:Total 6.2 MB/s | 31 MB 00:05 2023-12-06T15:28:35.833 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:28:35.872 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:28:35.873 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:28:36.204 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:28:36.207 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:28:37.324 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:28:37.546 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-06T15:28:37.731 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-06T15:28:37.914 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-06T15:28:38.162 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-06T15:28:38.370 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-06T15:28:38.549 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-06T15:28:38.710 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-06T15:28:38.896 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-06T15:28:39.093 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-06T15:28:39.340 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-06T15:28:39.552 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-06T15:28:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-06T15:28:40.091 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephsqlite-2:16.2.14-335.gd03fca4f.el8.x86_64 5/6 2023-12-06T15:28:40.136 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-06T15:28:40.364 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-2:16.2.14-335.gd03fca4f.el8.x86_64 6/6 2023-12-06T15:28:40.384 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-06T15:28:40.597 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-06T15:28:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-test-2:16.2.14-335.gd03fca4f.el8.x86_64 6/6 2023-12-06T15:28:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-2:16.2.14-335.gd03fca4f.el8.x86_64 1/6 2023-12-06T15:28:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephsqlite-2:16.2.14-335.gd03fca4f.el8.x86_64 2/6 2023-12-06T15:28:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-06T15:28:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-06T15:28:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-06T15:28:40.814 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-06T15:28:41.057 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-06T15:28:41.357 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-06T15:28:41.414 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-06T15:28:41.414 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:28:41.414 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:28:41.415 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:41.415 INFO:teuthology.orchestra.run.smithi176.stdout: jq-1.6-8.el8.x86_64 2023-12-06T15:28:41.415 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:41.415 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-12-06T15:28:41.415 INFO:teuthology.orchestra.run.smithi176.stdout: socat-1.7.4.1-1.el8.x86_64 2023-12-06T15:28:41.415 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-12-06T15:28:41.415 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:28:41.415 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:28:41.545 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-06T15:28:41.651 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph 2023-12-06T15:28:41.750 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-06T15:28:41.953 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-06T15:28:42.230 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-06T15:28:42.435 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-06T15:28:42.508 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:05 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:28:42.660 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-06T15:28:42.988 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-06T15:28:43.284 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-06T15:28:43.621 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-335.gd03fca4f.el8. 27/32 2023-12-06T15:28:43.827 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:16.2.14-335.gd03fca4f.el8.x86_64 28/32 2023-12-06T15:28:44.191 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:28:44.194 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:28:44.194 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-06T15:28:44.194 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:28:44.194 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:28:44.195 INFO:teuthology.orchestra.run.smithi176.stdout: ceph x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 6.4 k 2023-12-06T15:28:44.195 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-06T15:28:44.195 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 2.4 M 2023-12-06T15:28:44.195 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 1.6 M 2023-12-06T15:28:44.195 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 245 k 2023-12-06T15:28:44.195 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 5.0 M 2023-12-06T15:28:44.195 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 18 M 2023-12-06T15:28:44.195 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-06T15:28:44.195 INFO:teuthology.orchestra.run.smithi176.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-06T15:28:44.195 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-06T15:28:44.195 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-06T15:28:44.195 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-06T15:28:44.196 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-06T15:28:44.197 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-06T15:28:44.197 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-06T15:28:44.197 INFO:teuthology.orchestra.run.smithi176.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-06T15:28:44.197 INFO:teuthology.orchestra.run.smithi176.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-06T15:28:44.197 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-06T15:28:44.197 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-06T15:28:44.197 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-06T15:28:44.197 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-06T15:28:44.197 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:28:44.197 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:28:44.197 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:28:44.197 INFO:teuthology.orchestra.run.smithi176.stdout:Install 32 Packages 2023-12-06T15:28:44.198 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:28:44.198 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 31 M 2023-12-06T15:28:44.198 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 101 M 2023-12-06T15:28:44.198 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:28:44.591 INFO:teuthology.orchestra.run.smithi176.stdout:(1/32): ceph-16.2.14-335.gd03fca4f.el8.x86_64.r 53 kB/s | 6.4 kB 00:00 2023-12-06T15:28:44.750 INFO:teuthology.orchestra.run.smithi176.stdout:(2/32): ceph-mgr-16.2.14-335.gd03fca4f.el8.x86_ 5.8 MB/s | 1.6 MB 00:00 2023-12-06T15:28:45.309 INFO:teuthology.orchestra.run.smithi176.stdout:(3/32): ceph-mds-16.2.14-335.gd03fca4f.el8.x86_ 2.9 MB/s | 2.4 MB 00:00 2023-12-06T15:28:45.314 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:16.2.14-335.gd03fca4f.el8.x86_64 28/32 2023-12-06T15:28:45.459 INFO:teuthology.orchestra.run.smithi176.stdout:(4/32): ceph-mgr-modules-core-16.2.14-335.gd03f 1.6 MB/s | 245 kB 00:00 2023-12-06T15:28:45.496 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:16.2.14-335.gd03fca4f.el8.x86_64 29/32 2023-12-06T15:28:45.676 INFO:teuthology.orchestra.run.smithi176.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 725 kB/s | 157 kB 00:00 2023-12-06T15:28:45.734 INFO:teuthology.orchestra.run.smithi176.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 665 kB/s | 39 kB 00:00 2023-12-06T15:28:45.818 INFO:teuthology.orchestra.run.smithi176.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.2 MB/s | 103 kB 00:00 2023-12-06T15:28:45.910 INFO:teuthology.orchestra.run.smithi176.stdout:(8/32): ceph-mon-16.2.14-335.gd03fca4f.el8.x86_ 3.8 MB/s | 5.0 MB 00:01 2023-12-06T15:28:45.921 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:16.2.14-335.gd03fca4f.el8.x86_64 29/32 2023-12-06T15:28:45.927 INFO:teuthology.orchestra.run.smithi176.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 525 kB/s | 57 kB 00:00 2023-12-06T15:28:46.128 INFO:teuthology.orchestra.run.smithi176.stdout:(10/32): ceph-osd-16.2.14-335.gd03fca4f.el8.x86 13 MB/s | 18 MB 00:01 2023-12-06T15:28:46.244 INFO:teuthology.orchestra.run.smithi176.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.3 MB/s | 457 kB 00:00 2023-12-06T15:28:46.515 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:16.2.14-335.gd03fca4f.el8.x86_64 30/32 2023-12-06T15:28:46.545 INFO:teuthology.orchestra.run.smithi176.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 403 kB/s | 249 kB 00:00 2023-12-06T15:28:46.637 INFO:teuthology.orchestra.run.smithi176.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 278 kB/s | 109 kB 00:00 2023-12-06T15:28:46.737 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:16.2.14-335.gd03fca4f.el8.x86_64 30/32 2023-12-06T15:28:46.753 INFO:teuthology.orchestra.run.smithi176.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 1.0 MB/s | 641 kB 00:00 2023-12-06T15:28:46.904 INFO:teuthology.orchestra.run.smithi176.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 121 kB/s | 44 kB 00:00 2023-12-06T15:28:46.998 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:16.2.14-335.gd03fca4f.el8.x86_64 31/32 2023-12-06T15:28:47.016 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:16.2.14-335.gd03fca4f.el8.x86_64 31/32 2023-12-06T15:28:47.112 INFO:teuthology.orchestra.run.smithi176.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 389 kB/s | 185 kB 00:00 2023-12-06T15:28:47.130 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:16.2.14-335.gd03fca4f.el8.x86_64 32/32 2023-12-06T15:28:47.187 INFO:teuthology.orchestra.run.smithi176.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.3 MB/s | 384 kB 00:00 2023-12-06T15:28:47.213 INFO:teuthology.orchestra.run.smithi176.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2023-12-06T15:28:47.272 INFO:teuthology.orchestra.run.smithi176.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 333 kB/s | 173 kB 00:00 2023-12-06T15:28:47.297 INFO:teuthology.orchestra.run.smithi176.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 161 kB/s | 18 kB 00:00 2023-12-06T15:28:47.322 INFO:teuthology.orchestra.run.smithi176.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 448 kB/s | 49 kB 00:00 2023-12-06T15:28:47.372 INFO:teuthology.orchestra.run.smithi176.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 585 kB/s | 59 kB 00:00 2023-12-06T15:28:47.406 INFO:teuthology.orchestra.run.smithi176.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2023-12-06T15:28:47.432 INFO:teuthology.orchestra.run.smithi176.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2023-12-06T15:28:47.473 INFO:teuthology.orchestra.run.smithi176.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 186 kB/s | 19 kB 00:00 2023-12-06T15:28:47.507 INFO:teuthology.orchestra.run.smithi176.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 237 kB/s | 24 kB 00:00 2023-12-06T15:28:47.525 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-2:16.2.14-335.gd03fca4f.el8.x86_64 32/32 2023-12-06T15:28:47.525 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:16.2.14-335.gd03fca4f.el8.x86_64 1/32 2023-12-06T15:28:47.525 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:16.2.14-335.gd03fca4f.el8.x86_64 2/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:16.2.14-335.gd03fca4f.el8.x86_64 3/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:16.2.14-335.gd03fca4f.el8.x86_64 4/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:16.2.14-335.gd03fca4f.el8.x86_64 5/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-335.gd03fca4f.el8. 6/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-06T15:28:47.526 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-06T15:28:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-06T15:28:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-06T15:28:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-06T15:28:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-06T15:28:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-06T15:28:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-06T15:28:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-06T15:28:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-06T15:28:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-06T15:28:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-06T15:28:47.527 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-06T15:28:47.528 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-06T15:28:47.528 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-06T15:28:47.528 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-06T15:28:47.532 INFO:teuthology.orchestra.run.smithi176.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2023-12-06T15:28:47.574 INFO:teuthology.orchestra.run.smithi176.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2023-12-06T15:28:47.633 INFO:teuthology.orchestra.run.smithi176.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2023-12-06T15:28:47.675 INFO:teuthology.orchestra.run.smithi176.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 851 kB/s | 85 kB 00:00 2023-12-06T15:28:47.733 INFO:teuthology.orchestra.run.smithi176.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2023-12-06T15:28:47.842 INFO:teuthology.orchestra.run.smithi176.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 812 kB/s | 251 kB 00:00 2023-12-06T15:28:47.843 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:28:47.843 INFO:teuthology.orchestra.run.smithi176.stdout:Total 8.6 MB/s | 31 MB 00:03 2023-12-06T15:28:47.925 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:28:47.963 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:28:47.963 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:28:48.271 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:28:48.273 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:28:48.650 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-06T15:28:48.650 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:48.650 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:28:48.650 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:48.650 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:48.650 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:48.650 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:28:48.650 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:48.651 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:28:48.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-06T15:28:48.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-06T15:28:48.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-06T15:28:48.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-06T15:28:48.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-06T15:28:48.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-06T15:28:48.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-06T15:28:48.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-06T15:28:48.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-06T15:28:48.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-06T15:28:48.651 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-2.6-1.el8.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-06T15:28:48.652 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-06T15:28:48.653 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-06T15:28:48.653 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:48.653 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:28:48.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-base 2023-12-06T15:28:49.471 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:28:49.677 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-06T15:28:49.680 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:21 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:28:49.905 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-06T15:28:50.112 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-06T15:28:50.409 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-06T15:28:50.643 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-06T15:28:50.822 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-06T15:28:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-06T15:28:51.151 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-base-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:28:51.252 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-06T15:28:51.371 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:28:51.373 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2023-12-06T15:28:51.373 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:28:51.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install cephadm 2023-12-06T15:28:51.510 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-06T15:28:51.765 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-06T15:28:51.953 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-06T15:28:52.219 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-06T15:28:52.334 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:24 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:28:52.640 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-06T15:28:52.933 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-06T15:28:53.137 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-06T15:28:53.412 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-06T15:28:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-06T15:28:54.021 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-06T15:28:54.025 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:28:54.027 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:28:54.027 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2023-12-06T15:28:54.027 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:28:54.027 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:28:54.027 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 85 k 2023-12-06T15:28:54.027 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:54.027 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:28:54.027 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:28:54.028 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2023-12-06T15:28:54.028 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:54.028 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 85 k 2023-12-06T15:28:54.028 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 352 k 2023-12-06T15:28:54.028 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:28:54.210 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-06T15:28:54.278 INFO:teuthology.orchestra.run.smithi204.stdout:cephadm-16.2.14-335.gd03fca4f.el8.noarch.rpm 341 kB/s | 85 kB 00:00 2023-12-06T15:28:54.278 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:28:54.279 INFO:teuthology.orchestra.run.smithi204.stdout:Total 339 kB/s | 85 kB 00:00 2023-12-06T15:28:54.279 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:28:54.286 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:28:54.286 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:28:54.288 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:28:54.291 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:28:54.339 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:28:54.448 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-06T15:28:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-06T15:28:54.858 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 1/1 2023-12-06T15:28:54.941 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-06T15:28:54.988 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 1/1 2023-12-06T15:28:55.191 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-06T15:28:55.214 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 1/1 2023-12-06T15:28:55.397 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-06T15:28:55.701 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-06T15:28:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 1/1 2023-12-06T15:28:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:55.883 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:28:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:28:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:55.884 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:28:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-06T15:28:56.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-immutable-object-cache 2023-12-06T15:28:56.422 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-335.gd03fca4f.el8. 27/32 2023-12-06T15:28:56.612 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-2:16.2.14-335.gd03fca4f.el8.x86_64 28/32 2023-12-06T15:28:56.952 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:28 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:28:58.098 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-2:16.2.14-335.gd03fca4f.el8.x86_64 28/32 2023-12-06T15:28:58.304 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-2:16.2.14-335.gd03fca4f.el8.x86_64 29/32 2023-12-06T15:28:58.621 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:28:58.621 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:28:58.621 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2023-12-06T15:28:58.622 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:28:58.622 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:28:58.622 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 161 k 2023-12-06T15:28:58.622 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:58.622 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:28:58.622 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:28:58.622 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2023-12-06T15:28:58.622 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:28:58.623 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 161 k 2023-12-06T15:28:58.623 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 470 k 2023-12-06T15:28:58.623 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:28:58.729 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-2:16.2.14-335.gd03fca4f.el8.x86_64 29/32 2023-12-06T15:28:58.938 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-2:16.2.14-335.gd03fca4f.el8.x86_64 30/32 2023-12-06T15:28:59.009 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-immutable-object-cache-16.2.14-335.gd03fca 417 kB/s | 161 kB 00:00 2023-12-06T15:28:59.010 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:28:59.010 INFO:teuthology.orchestra.run.smithi204.stdout:Total 416 kB/s | 161 kB 00:00 2023-12-06T15:28:59.010 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:28:59.021 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:28:59.021 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:28:59.069 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:28:59.071 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:28:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-2:16.2.14-335.gd03fca4f.el8.x86_64 30/32 2023-12-06T15:28:59.184 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:28:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-335.gd03fca4f. 1/1 2023-12-06T15:28:59.721 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-2:16.2.14-335.gd03fca4f.el8.x86_64 31/32 2023-12-06T15:28:59.740 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-2:16.2.14-335.gd03fca4f.el8.x86_64 31/32 2023-12-06T15:28:59.745 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-335.gd03fca4f. 1/1 2023-12-06T15:28:59.870 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-2:16.2.14-335.gd03fca4f.el8.x86_64 32/32 2023-12-06T15:29:00.297 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-2:16.2.14-335.gd03fca4f.el8.x86_64 32/32 2023-12-06T15:29:00.297 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-2:16.2.14-335.gd03fca4f.el8.x86_64 1/32 2023-12-06T15:29:00.297 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-2:16.2.14-335.gd03fca4f.el8.x86_64 2/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-2:16.2.14-335.gd03fca4f.el8.x86_64 3/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-2:16.2.14-335.gd03fca4f.el8.x86_64 4/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-2:16.2.14-335.gd03fca4f.el8.x86_64 5/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-335.gd03fca4f.el8. 6/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-06T15:29:00.298 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-06T15:29:00.299 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-06T15:29:00.434 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-335.gd03fca4f. 1/1 2023-12-06T15:29:00.434 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:00.435 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:29:00.435 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:29:00.435 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:00.435 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:29:00.635 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr 2023-12-06T15:29:01.431 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-06T15:29:01.431 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:01.431 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:29:01.431 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:29:01.431 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:29:01.431 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:29:01.431 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:01.431 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:29:01.431 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:29:01.431 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-06T15:29:01.431 INFO:teuthology.orchestra.run.smithi176.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-06T15:29:01.431 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend-2.6-1.el8.noarch 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-06T15:29:01.432 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-06T15:29:01.433 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-06T15:29:01.433 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-06T15:29:01.433 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-06T15:29:01.433 INFO:teuthology.orchestra.run.smithi176.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-06T15:29:01.433 INFO:teuthology.orchestra.run.smithi176.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-06T15:29:01.433 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-06T15:29:01.433 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-06T15:29:01.433 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-06T15:29:01.433 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-06T15:29:01.433 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:01.433 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:29:01.485 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:33 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:29:01.627 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-base 2023-12-06T15:29:02.482 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:25 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:29:02.952 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-mgr-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:29:03.182 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:29:03.183 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2023-12-06T15:29:03.183 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:29:03.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-dashboard 2023-12-06T15:29:03.930 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-base-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:29:04.124 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:36 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:29:04.150 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:29:04.152 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-06T15:29:04.152 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:29:04.261 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install cephadm 2023-12-06T15:29:05.109 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:28 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:29:05.802 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:29:05.803 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 3.1 M 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 24 k 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 15 k 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:05.804 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2023-12-06T15:29:05.805 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:05.805 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 3.4 M 2023-12-06T15:29:05.805 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 22 M 2023-12-06T15:29:05.805 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:29:06.805 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:29:06.806 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:06.806 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-06T15:29:06.807 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:06.807 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:29:06.807 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 85 k 2023-12-06T15:29:06.807 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:06.807 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:29:06.807 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:06.807 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-06T15:29:06.807 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:06.807 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 85 k 2023-12-06T15:29:06.807 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 352 k 2023-12-06T15:29:06.807 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:29:06.973 INFO:teuthology.orchestra.run.smithi176.stdout:cephadm-16.2.14-335.gd03fca4f.el8.noarch.rpm 514 kB/s | 85 kB 00:00 2023-12-06T15:29:06.974 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:06.974 INFO:teuthology.orchestra.run.smithi176.stdout:Total 510 kB/s | 85 kB 00:00 2023-12-06T15:29:06.974 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:29:06.981 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:29:06.981 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:29:06.983 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:29:06.986 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:29:07.045 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:29:07.195 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): ceph-grafana-dashboards-16.2.14-335.gd03 172 kB/s | 24 kB 00:00 2023-12-06T15:29:07.295 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): ceph-prometheus-alerts-16.2.14-335.gd03f 61 kB/s | 15 kB 00:00 2023-12-06T15:29:07.478 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 183 kB/s | 34 kB 00:00 2023-12-06T15:29:07.511 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 1/1 2023-12-06T15:29:07.528 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 128 kB/s | 43 kB 00:00 2023-12-06T15:29:07.621 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): ceph-mgr-dashboard-16.2.14-335.gd03fca4f 5.5 MB/s | 3.1 MB 00:00 2023-12-06T15:29:07.646 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2023-12-06T15:29:07.649 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:07.649 INFO:teuthology.orchestra.run.smithi204.stdout:Total 1.9 MB/s | 3.4 MB 00:01 2023-12-06T15:29:07.656 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 1/1 2023-12-06T15:29:07.661 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:29:07.672 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:29:07.672 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:29:07.830 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:29:07.833 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:29:07.884 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 1/1 2023-12-06T15:29:08.149 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:29:08.334 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-06T15:29:08.554 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-06T15:29:08.568 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 1/1 2023-12-06T15:29:08.568 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:08.568 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:29:08.568 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:08.568 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:08.568 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:29:08.730 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-06T15:29:08.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-immutable-object-cache 2023-12-06T15:29:08.867 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-335.gd03fca4f.el8.n 4/6 2023-12-06T15:29:09.649 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:32 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:29:09.874 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-335.gd03fca4f.el8. 5/6 2023-12-06T15:29:10.123 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-335.gd03fca4f.el8.noarc 6/6 2023-12-06T15:29:10.395 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-335.gd03fca4f.el8.noarc 6/6 2023-12-06T15:29:10.396 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-335.gd03fca4f.el8. 1/6 2023-12-06T15:29:10.396 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-335.gd03fca4f.el8.noarc 2/6 2023-12-06T15:29:10.396 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-335.gd03fca4f.el8.n 3/6 2023-12-06T15:29:10.396 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-06T15:29:10.396 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-06T15:29:11.081 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-06T15:29:11.081 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:11.081 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:29:11.081 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:11.081 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:11.082 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:11.082 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-06T15:29:11.082 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-06T15:29:11.082 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-06T15:29:11.082 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:11.082 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:29:11.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-06T15:29:11.341 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:29:11.341 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:11.341 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2023-12-06T15:29:11.341 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:11.342 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:29:11.342 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 161 k 2023-12-06T15:29:11.342 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:11.342 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:29:11.342 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:11.342 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-06T15:29:11.342 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:11.342 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 161 k 2023-12-06T15:29:11.343 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 470 k 2023-12-06T15:29:11.343 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:29:11.636 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-immutable-object-cache-16.2.14-335.gd03fca 549 kB/s | 161 kB 00:00 2023-12-06T15:29:11.636 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:11.637 INFO:teuthology.orchestra.run.smithi176.stdout:Total 547 kB/s | 161 kB 00:00 2023-12-06T15:29:11.637 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:29:11.648 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:29:11.648 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:29:11.695 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:29:11.698 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:29:11.808 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:29:12.010 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-335.gd03fca4f. 1/1 2023-12-06T15:29:12.076 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:44 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:29:12.445 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-335.gd03fca4f. 1/1 2023-12-06T15:29:13.126 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-335.gd03fca4f. 1/1 2023-12-06T15:29:13.126 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:13.126 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:29:13.126 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:29:13.126 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:13.126 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:29:13.321 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr 2023-12-06T15:29:13.783 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:29:13.784 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================================ 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================================ 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 8.6 M 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-12-06T15:29:13.785 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:13.786 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:29:13.786 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================================ 2023-12-06T15:29:13.786 INFO:teuthology.orchestra.run.smithi204.stdout:Install 8 Packages 2023-12-06T15:29:13.786 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:13.787 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 37 M 2023-12-06T15:29:13.787 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 210 M 2023-12-06T15:29:13.787 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:29:14.168 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:37 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:29:15.631 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-mgr-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:29:15.856 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:29:15.857 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-06T15:29:15.857 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:29:15.877 INFO:teuthology.orchestra.run.smithi204.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 2.8 MB/s | 4.7 MB 00:01 2023-12-06T15:29:15.953 INFO:teuthology.orchestra.run.smithi204.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 2.7 MB/s | 4.9 MB 00:01 2023-12-06T15:29:15.973 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-dashboard 2023-12-06T15:29:16.045 INFO:teuthology.orchestra.run.smithi204.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-12-06T15:29:16.503 INFO:teuthology.orchestra.run.smithi204.stdout:(4/8): ceph-mgr-diskprediction-local-16.2.14-33 3.7 MB/s | 8.6 MB 00:02 2023-12-06T15:29:16.562 INFO:teuthology.orchestra.run.smithi204.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.5 MB/s | 3.7 MB 00:00 2023-12-06T15:29:16.818 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:39 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:29:17.114 INFO:teuthology.orchestra.run.smithi204.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 312 kB/s | 172 kB 00:00 2023-12-06T15:29:17.265 INFO:teuthology.orchestra.run.smithi204.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 848 kB/s | 645 kB 00:00 2023-12-06T15:29:17.982 INFO:teuthology.orchestra.run.smithi204.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 7.1 MB/s | 14 MB 00:01 2023-12-06T15:29:17.982 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:17.982 INFO:teuthology.orchestra.run.smithi204.stdout:Total 8.7 MB/s | 37 MB 00:04 2023-12-06T15:29:18.218 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:29:18.236 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:29:18.236 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:29:18.507 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:29:18.508 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:18.508 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-06T15:29:18.508 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 3.1 M 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 24 k 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 15 k 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2023-12-06T15:29:18.509 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:18.510 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 3.4 M 2023-12-06T15:29:18.510 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 22 M 2023-12-06T15:29:18.510 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:29:18.573 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:29:18.576 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:29:19.072 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): ceph-prometheus-alerts-16.2.14-335.gd03f 51 kB/s | 15 kB 00:00 2023-12-06T15:29:19.097 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): ceph-grafana-dashboards-16.2.14-335.gd03 77 kB/s | 24 kB 00:00 2023-12-06T15:29:19.158 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:29:19.173 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): ceph-mgr-dashboard-16.2.14-335.gd03fca4f 8.0 MB/s | 3.1 MB 00:00 2023-12-06T15:29:19.302 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-06T15:29:19.443 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-06T15:29:19.515 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 96 kB/s | 43 kB 00:00 2023-12-06T15:29:19.565 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 72 kB/s | 34 kB 00:00 2023-12-06T15:29:19.605 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-06T15:29:20.065 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 220 kB/s | 196 kB 00:00 2023-12-06T15:29:20.068 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:20.069 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.2 MB/s | 3.4 MB 00:01 2023-12-06T15:29:20.080 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:29:20.091 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:29:20.091 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:29:20.204 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-06T15:29:20.241 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:29:20.244 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:29:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:29:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-06T15:29:20.764 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-06T15:29:21.018 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-06T15:29:21.194 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-06T15:29:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-06T15:29:21.348 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-335.gd03fca4f.el8.n 4/6 2023-12-06T15:29:21.468 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-06T15:29:22.164 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-06T15:29:22.374 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-335.gd03fca4f.el8. 5/6 2023-12-06T15:29:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-06T15:29:22.674 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-335.gd03fca4f.el8.noarc 6/6 2023-12-06T15:29:22.792 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-06T15:29:22.987 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-335.gd03fca4f.el8.noarc 6/6 2023-12-06T15:29:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-335.gd03fca4f.el8. 1/6 2023-12-06T15:29:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-335.gd03fca4f.el8.noarc 2/6 2023-12-06T15:29:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-335.gd03fca4f.el8.n 3/6 2023-12-06T15:29:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-06T15:29:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-06T15:29:23.686 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-06T15:29:23.686 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:23.686 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:29:23.686 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:23.686 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:23.686 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:23.686 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-06T15:29:23.686 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-06T15:29:23.687 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-06T15:29:23.687 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:23.687 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:29:23.810 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-06T15:29:24.550 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-06T15:29:24.666 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:47 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:29:25.842 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-12-06T15:29:26.042 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-335.gd03fca4 8/8 2023-12-06T15:29:26.334 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:29:26.336 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================================ 2023-12-06T15:29:26.336 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-06T15:29:26.336 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================================ 2023-12-06T15:29:26.336 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:29:26.336 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 8.6 M 2023-12-06T15:29:26.336 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-06T15:29:26.336 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-06T15:29:26.337 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-06T15:29:26.337 INFO:teuthology.orchestra.run.smithi176.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-06T15:29:26.337 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-06T15:29:26.337 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-06T15:29:26.337 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-06T15:29:26.337 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-12-06T15:29:26.337 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:26.337 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:29:26.337 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================================ 2023-12-06T15:29:26.337 INFO:teuthology.orchestra.run.smithi176.stdout:Install 8 Packages 2023-12-06T15:29:26.337 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:26.338 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 37 M 2023-12-06T15:29:26.338 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 210 M 2023-12-06T15:29:26.338 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:29:26.363 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-335.gd03fca4 8/8 2023-12-06T15:29:26.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-335.gd03fca4 1/8 2023-12-06T15:29:26.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-06T15:29:26.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-06T15:29:26.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-06T15:29:26.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-06T15:29:26.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-12-06T15:29:26.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-06T15:29:27.082 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-06T15:29:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:27.083 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:29:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-06T15:29:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-06T15:29:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-06T15:29:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-06T15:29:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-06T15:29:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-06T15:29:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-12-06T15:29:27.083 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:27.083 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:29:27.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-rook 2023-12-06T15:29:28.056 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:00 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:29:28.738 INFO:teuthology.orchestra.run.smithi176.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 2.4 MB/s | 4.9 MB 00:02 2023-12-06T15:29:29.072 INFO:teuthology.orchestra.run.smithi176.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 2.0 MB/s | 4.7 MB 00:02 2023-12-06T15:29:29.180 INFO:teuthology.orchestra.run.smithi176.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2023-12-06T15:29:29.746 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:29:29.747 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:29.747 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2023-12-06T15:29:29.747 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 27 k 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-06T15:29:29.748 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-06T15:29:29.749 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-06T15:29:29.749 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:29.749 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:29:29.749 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:29.749 INFO:teuthology.orchestra.run.smithi204.stdout:Install 13 Packages 2023-12-06T15:29:29.749 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:29.749 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 1.9 M 2023-12-06T15:29:29.749 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 24 M 2023-12-06T15:29:29.749 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:29:29.864 INFO:teuthology.orchestra.run.smithi176.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.3 MB/s | 3.7 MB 00:01 2023-12-06T15:29:30.189 INFO:teuthology.orchestra.run.smithi204.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 178 kB/s | 27 kB 00:00 2023-12-06T15:29:30.205 INFO:teuthology.orchestra.run.smithi204.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 120 kB/s | 20 kB 00:00 2023-12-06T15:29:30.223 INFO:teuthology.orchestra.run.smithi204.stdout:(3/13): ceph-mgr-rook-16.2.14-335.gd03fca4f.el8 149 kB/s | 27 kB 00:00 2023-12-06T15:29:30.373 INFO:teuthology.orchestra.run.smithi204.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 756 kB/s | 126 kB 00:00 2023-12-06T15:29:30.406 INFO:teuthology.orchestra.run.smithi204.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 549 kB/s | 110 kB 00:00 2023-12-06T15:29:30.414 INFO:teuthology.orchestra.run.smithi176.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2023-12-06T15:29:30.522 INFO:teuthology.orchestra.run.smithi176.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.6 MB/s | 172 kB 00:00 2023-12-06T15:29:30.624 INFO:teuthology.orchestra.run.smithi204.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 387 kB/s | 155 kB 00:00 2023-12-06T15:29:30.640 INFO:teuthology.orchestra.run.smithi204.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 141 kB/s | 33 kB 00:00 2023-12-06T15:29:30.656 INFO:teuthology.orchestra.run.smithi176.stdout:(7/8): ceph-mgr-diskprediction-local-16.2.14-33 2.2 MB/s | 8.6 MB 00:03 2023-12-06T15:29:30.716 INFO:teuthology.orchestra.run.smithi204.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 138 kB/s | 43 kB 00:00 2023-12-06T15:29:30.783 INFO:teuthology.orchestra.run.smithi204.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 572 kB/s | 81 kB 00:00 2023-12-06T15:29:30.841 INFO:teuthology.orchestra.run.smithi204.stdout:(10/13): python3-certifi-2018.10.15-7.el8.noarc 75 kB/s | 16 kB 00:00 2023-12-06T15:29:30.883 INFO:teuthology.orchestra.run.smithi204.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 672 kB/s | 64 kB 00:00 2023-12-06T15:29:30.975 INFO:teuthology.orchestra.run.smithi204.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 461 kB/s | 61 kB 00:00 2023-12-06T15:29:31.251 INFO:teuthology.orchestra.run.smithi204.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.2 MB/s | 1.2 MB 00:00 2023-12-06T15:29:31.254 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:31.255 INFO:teuthology.orchestra.run.smithi204.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-12-06T15:29:31.315 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:29:31.329 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:29:31.329 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:29:31.417 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:29:31.420 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:29:31.909 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:29:32.194 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-06T15:29:32.391 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-06T15:29:32.609 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-06T15:29:32.840 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-06T15:29:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-06T15:29:33.156 INFO:teuthology.orchestra.run.smithi176.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 3.4 MB/s | 14 MB 00:03 2023-12-06T15:29:33.158 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:33.158 INFO:teuthology.orchestra.run.smithi176.stdout:Total 5.4 MB/s | 37 MB 00:06 2023-12-06T15:29:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-06T15:29:33.393 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:29:33.411 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:29:33.411 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:29:33.622 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-06T15:29:33.730 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:29:33.733 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:29:33.854 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-06T15:29:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:29:34.490 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-06T15:29:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-06T15:29:34.815 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-06T15:29:34.901 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-06T15:29:35.189 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-06T15:29:35.390 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-06T15:29:35.410 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-06T15:29:35.573 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-06T15:29:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-06T15:29:35.757 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-rook-2:16.2.14-335.gd03fca4f.el8.noarch 13/13 2023-12-06T15:29:36.057 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-335.gd03fca4f.el8.noarch 13/13 2023-12-06T15:29:36.057 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-rook-2:16.2.14-335.gd03fca4f.el8.noarch 1/13 2023-12-06T15:29:36.057 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-06T15:29:36.057 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-06T15:29:36.058 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-06T15:29:36.058 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-06T15:29:36.058 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-06T15:29:36.058 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-06T15:29:36.058 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-06T15:29:36.058 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-06T15:29:36.058 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-06T15:29:36.058 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-06T15:29:36.058 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-06T15:29:36.238 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-06T15:29:36.431 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-06T15:29:36.806 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-06T15:29:36.806 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:36.806 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:29:36.806 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:36.806 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-06T15:29:36.806 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-06T15:29:36.806 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-06T15:29:36.806 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-06T15:29:36.806 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-06T15:29:36.806 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-06T15:29:36.806 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-06T15:29:36.806 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-06T15:29:36.807 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-06T15:29:36.807 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-06T15:29:36.807 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-06T15:29:36.807 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-06T15:29:36.807 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:36.807 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:29:36.926 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-cephadm 2023-12-06T15:29:37.133 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-06T15:29:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-06T15:29:37.782 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:09 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:29:37.807 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-06T15:29:39.471 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:29:39.471 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2023-12-06T15:29:39.472 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:39.472 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:29:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 116 k 2023-12-06T15:29:39.472 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2023-12-06T15:29:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-06T15:29:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-06T15:29:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-06T15:29:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-06T15:29:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-06T15:29:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:39.473 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:29:39.473 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:39.473 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2023-12-06T15:29:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:39.474 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 5.8 M 2023-12-06T15:29:39.474 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 24 M 2023-12-06T15:29:39.474 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:29:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-06T15:29:39.759 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): ceph-mgr-cephadm-16.2.14-335.gd03fca4f.e 631 kB/s | 116 kB 00:00 2023-12-06T15:29:39.859 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.8 MB/s | 538 kB 00:00 2023-12-06T15:29:40.143 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 55 kB/s | 21 kB 00:00 2023-12-06T15:29:40.260 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.0 MB/s | 4.8 MB 00:00 2023-12-06T15:29:40.327 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 266 kB/s | 49 kB 00:00 2023-12-06T15:29:40.435 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 622 kB/s | 355 kB 00:00 2023-12-06T15:29:40.438 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:40.438 INFO:teuthology.orchestra.run.smithi204.stdout:Total 6.1 MB/s | 5.8 MB 00:00 2023-12-06T15:29:40.489 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:29:40.499 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:29:40.499 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:29:40.608 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:29:40.611 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:29:40.873 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-12-06T15:29:40.945 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:29:41.097 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-335.gd03fca4 8/8 2023-12-06T15:29:41.366 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-06T15:29:41.417 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-335.gd03fca4 8/8 2023-12-06T15:29:41.417 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-335.gd03fca4 1/8 2023-12-06T15:29:41.417 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-06T15:29:41.418 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-06T15:29:41.418 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-06T15:29:41.418 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-06T15:29:41.418 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-12-06T15:29:41.418 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-06T15:29:41.584 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-06T15:29:42.153 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-06T15:29:42.153 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:42.153 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:29:42.153 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:42.153 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-06T15:29:42.153 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-06T15:29:42.154 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-06T15:29:42.154 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-06T15:29:42.154 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-06T15:29:42.154 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-06T15:29:42.154 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-12-06T15:29:42.154 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:42.154 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:29:42.272 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-rook 2023-12-06T15:29:42.507 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-06T15:29:42.853 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-06T15:29:43.107 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-06T15:29:43.126 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:06 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:29:43.298 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 6/6 2023-12-06T15:29:43.585 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 6/6 2023-12-06T15:29:43.585 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 1/6 2023-12-06T15:29:43.585 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-06T15:29:43.586 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-06T15:29:43.586 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-06T15:29:43.586 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-06T15:29:44.279 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-06T15:29:44.279 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:44.279 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:29:44.279 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:44.279 INFO:teuthology.orchestra.run.smithi204.stdout: python3-apipkg-1.5-6.el8.noarch 2023-12-06T15:29:44.279 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel-2.5.1-7.el8.noarch 2023-12-06T15:29:44.279 INFO:teuthology.orchestra.run.smithi204.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-12-06T15:29:44.279 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-12-06T15:29:44.279 INFO:teuthology.orchestra.run.smithi204.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-12-06T15:29:44.279 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:44.280 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:29:44.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-fuse 2023-12-06T15:29:44.785 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:29:44.786 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:44.786 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-06T15:29:44.786 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 27 k 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-06T15:29:44.787 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:44.788 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:29:44.788 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:44.788 INFO:teuthology.orchestra.run.smithi176.stdout:Install 13 Packages 2023-12-06T15:29:44.788 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:44.788 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 1.9 M 2023-12-06T15:29:44.788 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 24 M 2023-12-06T15:29:44.788 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:29:45.212 INFO:teuthology.orchestra.run.smithi176.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 192 kB/s | 27 kB 00:00 2023-12-06T15:29:45.229 INFO:teuthology.orchestra.run.smithi176.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 129 kB/s | 20 kB 00:00 2023-12-06T15:29:45.241 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:17 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:29:45.245 INFO:teuthology.orchestra.run.smithi176.stdout:(3/13): ceph-mgr-rook-16.2.14-335.gd03fca4f.el8 159 kB/s | 27 kB 00:00 2023-12-06T15:29:45.371 INFO:teuthology.orchestra.run.smithi176.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 890 kB/s | 126 kB 00:00 2023-12-06T15:29:45.388 INFO:teuthology.orchestra.run.smithi176.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 693 kB/s | 110 kB 00:00 2023-12-06T15:29:45.563 INFO:teuthology.orchestra.run.smithi176.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 188 kB/s | 33 kB 00:00 2023-12-06T15:29:45.630 INFO:teuthology.orchestra.run.smithi176.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 244 kB/s | 16 kB 00:00 2023-12-06T15:29:45.657 INFO:teuthology.orchestra.run.smithi176.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 378 kB/s | 155 kB 00:00 2023-12-06T15:29:45.714 INFO:teuthology.orchestra.run.smithi176.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 971 kB/s | 81 kB 00:00 2023-12-06T15:29:45.739 INFO:teuthology.orchestra.run.smithi176.stdout:(10/13): python3-requests-oauthlib-1.0.0-1.el8. 121 kB/s | 43 kB 00:00 2023-12-06T15:29:45.773 INFO:teuthology.orchestra.run.smithi176.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 1.2 MB/s | 64 kB 00:00 2023-12-06T15:29:45.915 INFO:teuthology.orchestra.run.smithi176.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 4.5 MB/s | 1.2 MB 00:00 2023-12-06T15:29:45.940 INFO:teuthology.orchestra.run.smithi176.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 306 kB/s | 61 kB 00:00 2023-12-06T15:29:45.942 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:45.942 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2023-12-06T15:29:46.000 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:29:46.016 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:29:46.016 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:29:46.101 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:29:46.104 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:29:46.630 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:29:46.932 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-06T15:29:47.036 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:29:47.036 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:47.036 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2023-12-06T15:29:47.036 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:47.036 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:29:47.037 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 910 k 2023-12-06T15:29:47.037 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2023-12-06T15:29:47.037 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-06T15:29:47.037 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:47.037 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:29:47.037 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:47.037 INFO:teuthology.orchestra.run.smithi204.stdout:Install 2 Packages 2023-12-06T15:29:47.037 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:47.037 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 993 k 2023-12-06T15:29:47.038 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 2.8 M 2023-12-06T15:29:47.038 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:29:47.128 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-06T15:29:47.330 INFO:teuthology.orchestra.run.smithi204.stdout:(1/2): ceph-fuse-16.2.14-335.gd03fca4f.el8.x86_ 3.5 MB/s | 910 kB 00:00 2023-12-06T15:29:47.356 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-06T15:29:47.405 INFO:teuthology.orchestra.run.smithi204.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 254 kB/s | 83 kB 00:00 2023-12-06T15:29:47.406 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:47.406 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.6 MB/s | 993 kB 00:00 2023-12-06T15:29:47.416 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:29:47.436 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:29:47.436 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:29:47.522 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:29:47.525 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:29:47.611 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-06T15:29:47.700 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:29:47.850 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-06T15:29:47.921 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-06T15:29:48.097 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 2/2 2023-12-06T15:29:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-06T15:29:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-06T15:29:48.482 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 2/2 2023-12-06T15:29:48.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 1/2 2023-12-06T15:29:48.717 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-06T15:29:49.189 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-06T15:29:49.189 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:49.189 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:29:49.189 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-06T15:29:49.189 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:49.189 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:29:49.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados-devel 2023-12-06T15:29:49.729 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-06T15:29:49.985 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-06T15:29:50.211 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-06T15:29:50.256 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:22 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:29:50.411 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-06T15:29:50.621 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-rook-2:16.2.14-335.gd03fca4f.el8.noarch 13/13 2023-12-06T15:29:50.915 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-335.gd03fca4f.el8.noarch 13/13 2023-12-06T15:29:50.915 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-rook-2:16.2.14-335.gd03fca4f.el8.noarch 1/13 2023-12-06T15:29:50.915 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-06T15:29:50.915 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-06T15:29:50.915 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-06T15:29:50.915 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-06T15:29:50.916 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-06T15:29:50.916 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-06T15:29:50.916 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-06T15:29:50.916 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-06T15:29:50.916 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-06T15:29:50.916 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-06T15:29:50.916 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-06T15:29:51.667 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-06T15:29:51.667 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:51.668 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:29:51.796 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-cephadm 2023-12-06T15:29:51.932 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:29:51.933 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:51.933 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2023-12-06T15:29:51.933 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:51.933 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:29:51.933 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 131 k 2023-12-06T15:29:51.933 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:51.933 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:29:51.933 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:51.933 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2023-12-06T15:29:51.933 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:51.934 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 131 k 2023-12-06T15:29:51.934 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 432 k 2023-12-06T15:29:51.934 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:29:52.649 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:15 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:29:52.732 INFO:teuthology.orchestra.run.smithi204.stdout:librados-devel-16.2.14-335.gd03fca4f.el8.x86_64 164 kB/s | 131 kB 00:00 2023-12-06T15:29:52.733 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:52.733 INFO:teuthology.orchestra.run.smithi204.stdout:Total 164 kB/s | 131 kB 00:00 2023-12-06T15:29:52.734 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:29:52.740 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:29:52.740 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:29:52.795 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:29:52.798 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:29:52.922 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:29:53.138 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:29:53.422 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:29:54.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:29:54.109 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:54.109 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:29:54.109 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:29:54.109 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:54.109 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:29:54.315 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs2 2023-12-06T15:29:54.336 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:29:54.337 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-06T15:29:54.337 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:54.337 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:29:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm noarch 2:16.2.14-335.gd03fca4f.el8 ceph-noarch 116 k 2023-12-06T15:29:54.337 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-06T15:29:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-06T15:29:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-06T15:29:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-06T15:29:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-06T15:29:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-06T15:29:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:54.338 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:29:54.338 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:29:54.338 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2023-12-06T15:29:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:54.338 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 5.8 M 2023-12-06T15:29:54.338 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 24 M 2023-12-06T15:29:54.338 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:29:54.752 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): ceph-mgr-cephadm-16.2.14-335.gd03fca4f.e 339 kB/s | 116 kB 00:00 2023-12-06T15:29:54.785 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.4 MB/s | 538 kB 00:00 2023-12-06T15:29:55.177 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 50 kB/s | 21 kB 00:00 2023-12-06T15:29:55.187 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:27 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:29:55.361 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 266 kB/s | 49 kB 00:00 2023-12-06T15:29:55.470 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 520 kB/s | 355 kB 00:00 2023-12-06T15:29:55.570 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 4.1 MB/s | 4.8 MB 00:01 2023-12-06T15:29:55.572 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:55.572 INFO:teuthology.orchestra.run.smithi176.stdout:Total 4.7 MB/s | 5.8 MB 00:01 2023-12-06T15:29:55.626 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:29:55.635 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:29:55.636 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:29:55.738 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:29:55.742 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:29:56.091 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:29:56.362 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-06T15:29:56.634 INFO:teuthology.orchestra.run.smithi204.stdout:Package libcephfs2-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:29:56.639 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-06T15:29:56.857 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:29:56.858 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2023-12-06T15:29:56.859 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:29:56.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs-devel 2023-12-06T15:29:57.607 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-06T15:29:57.824 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:29 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:29:57.972 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-06T15:29:58.227 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-06T15:29:58.419 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 6/6 2023-12-06T15:29:58.726 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 6/6 2023-12-06T15:29:58.726 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 1/6 2023-12-06T15:29:58.726 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-06T15:29:58.726 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-06T15:29:58.726 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-06T15:29:58.726 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-06T15:29:59.412 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-06T15:29:59.412 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:59.413 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:29:59.413 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm-2:16.2.14-335.gd03fca4f.el8.noarch 2023-12-06T15:29:59.413 INFO:teuthology.orchestra.run.smithi176.stdout: python3-apipkg-1.5-6.el8.noarch 2023-12-06T15:29:59.413 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel-2.5.1-7.el8.noarch 2023-12-06T15:29:59.413 INFO:teuthology.orchestra.run.smithi176.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-12-06T15:29:59.413 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-12-06T15:29:59.413 INFO:teuthology.orchestra.run.smithi176.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-12-06T15:29:59.413 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:29:59.413 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:29:59.504 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:29:59.505 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:59.505 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2023-12-06T15:29:59.505 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:59.505 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:29:59.505 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 25 k 2023-12-06T15:29:59.505 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:59.505 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:29:59.505 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:29:59.505 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2023-12-06T15:29:59.505 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:29:59.506 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 25 k 2023-12-06T15:29:59.506 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 107 k 2023-12-06T15:29:59.506 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:29:59.522 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-fuse 2023-12-06T15:29:59.725 INFO:teuthology.orchestra.run.smithi204.stdout:libcephfs-devel-16.2.14-335.gd03fca4f.el8.x86_6 117 kB/s | 25 kB 00:00 2023-12-06T15:29:59.726 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:29:59.726 INFO:teuthology.orchestra.run.smithi204.stdout:Total 116 kB/s | 25 kB 00:00 2023-12-06T15:29:59.726 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:29:59.729 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:29:59.729 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:29:59.731 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:29:59.734 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:29:59.788 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:30:00.003 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:00.276 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:00.373 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:23 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:00.937 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:00.937 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:00.937 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:30:00.937 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:30:00.938 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:00.938 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:30:01.066 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados2 2023-12-06T15:30:01.922 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:33 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:30:02.057 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:02.058 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:02.058 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-06T15:30:02.058 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:02.058 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:30:02.058 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 910 k 2023-12-06T15:30:02.058 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-06T15:30:02.058 INFO:teuthology.orchestra.run.smithi176.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-06T15:30:02.058 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:02.058 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:30:02.059 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:02.059 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2023-12-06T15:30:02.059 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:02.059 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 993 k 2023-12-06T15:30:02.059 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 2.8 M 2023-12-06T15:30:02.059 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:30:03.377 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:03.600 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:30:03.601 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2023-12-06T15:30:03.602 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:30:03.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librbd1 2023-12-06T15:30:04.554 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:36 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:30:06.011 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:06.236 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:30:06.237 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2023-12-06T15:30:06.237 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:30:06.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rados 2023-12-06T15:30:07.220 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:39 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:30:08.675 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rados-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:08.898 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:30:08.899 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2023-12-06T15:30:08.899 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:30:09.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rgw 2023-12-06T15:30:09.868 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:41 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:30:10.392 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 117 kB/s | 83 kB 00:00 2023-12-06T15:30:10.656 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-fuse-16.2.14-335.gd03fca4f.el8.x86_ 934 kB/s | 910 kB 00:00 2023-12-06T15:30:10.657 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:30:10.657 INFO:teuthology.orchestra.run.smithi176.stdout:Total 116 kB/s | 993 kB 00:08 2023-12-06T15:30:10.669 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:30:10.691 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:30:10.691 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:30:10.784 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:30:10.787 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:30:11.327 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rgw-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:11.551 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:30:11.552 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2023-12-06T15:30:11.552 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:30:11.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-cephfs 2023-12-06T15:30:11.849 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:30:12.157 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-06T15:30:12.358 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 2/2 2023-12-06T15:30:12.541 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:44 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:30:12.761 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 2/2 2023-12-06T15:30:12.761 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 1/2 2023-12-06T15:30:13.467 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-06T15:30:13.467 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:13.467 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:30:13.467 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-06T15:30:13.467 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:13.468 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:13.695 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados-devel 2023-12-06T15:30:13.997 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-cephfs-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:14.222 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:30:14.223 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2023-12-06T15:30:14.223 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:30:14.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rbd 2023-12-06T15:30:14.568 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:37 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:15.374 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:47 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:30:16.248 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:16.249 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:16.249 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2023-12-06T15:30:16.250 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:16.250 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:30:16.250 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 131 k 2023-12-06T15:30:16.250 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:16.250 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:30:16.250 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:16.250 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-06T15:30:16.250 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:16.250 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 131 k 2023-12-06T15:30:16.250 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 432 k 2023-12-06T15:30:16.251 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:30:16.444 INFO:teuthology.orchestra.run.smithi176.stdout:librados-devel-16.2.14-335.gd03fca4f.el8.x86_64 678 kB/s | 131 kB 00:00 2023-12-06T15:30:16.444 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:30:16.444 INFO:teuthology.orchestra.run.smithi176.stdout:Total 674 kB/s | 131 kB 00:00 2023-12-06T15:30:16.444 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:30:16.450 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:30:16.450 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:30:16.500 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:30:16.503 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:30:16.626 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:30:16.833 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rbd-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:16.899 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:17.056 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:30:17.057 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2023-12-06T15:30:17.057 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:30:17.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-fuse 2023-12-06T15:30:17.234 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:17.945 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:17.945 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:17.945 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:30:17.945 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:30:17.945 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:17.945 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:18.032 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:50 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:30:18.148 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs2 2023-12-06T15:30:19.012 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:42 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:19.730 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:30:19.731 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:30:19.731 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2023-12-06T15:30:19.731 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:30:19.731 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:30:19.731 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 88 k 2023-12-06T15:30:19.731 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:19.731 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:30:19.731 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:30:19.731 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2023-12-06T15:30:19.732 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:19.732 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 88 k 2023-12-06T15:30:19.732 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 240 k 2023-12-06T15:30:19.732 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:30:20.032 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-fuse-16.2.14-335.gd03fca4f.el8.x86_64.rpm 295 kB/s | 88 kB 00:00 2023-12-06T15:30:20.033 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:30:20.033 INFO:teuthology.orchestra.run.smithi204.stdout:Total 293 kB/s | 88 kB 00:00 2023-12-06T15:30:20.033 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:30:20.043 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:30:20.043 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:30:20.093 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:30:20.096 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:30:20.205 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:30:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:20.475 INFO:teuthology.orchestra.run.smithi176.stdout:Package libcephfs2-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:20.692 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:20.705 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:20.706 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-06T15:30:20.706 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:20.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs-devel 2023-12-06T15:30:21.385 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:21.386 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:21.386 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:30:21.386 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:30:21.386 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:21.386 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:30:21.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-mirror 2023-12-06T15:30:21.669 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:44 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:22.457 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:54 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:30:23.342 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 25 k 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 25 k 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 107 k 2023-12-06T15:30:23.343 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:30:23.558 INFO:teuthology.orchestra.run.smithi176.stdout:libcephfs-devel-16.2.14-335.gd03fca4f.el8.x86_6 119 kB/s | 25 kB 00:00 2023-12-06T15:30:23.558 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:30:23.559 INFO:teuthology.orchestra.run.smithi176.stdout:Total 119 kB/s | 25 kB 00:00 2023-12-06T15:30:23.559 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:30:23.562 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:30:23.562 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:30:23.564 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:30:23.567 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:30:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:30:23.839 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:24.105 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:24.139 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:30:24.139 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:30:24.139 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2023-12-06T15:30:24.139 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:30:24.139 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:30:24.139 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 3.7 M 2023-12-06T15:30:24.139 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:24.140 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:30:24.140 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:30:24.140 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2023-12-06T15:30:24.140 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:24.140 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 3.7 M 2023-12-06T15:30:24.141 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 15 M 2023-12-06T15:30:24.141 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:30:24.792 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:24.793 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:24.793 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:30:24.793 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:30:24.793 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:24.793 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:24.908 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados2 2023-12-06T15:30:25.194 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-mirror-16.2.14-335.gd03fca4f.el8.x86_64.rpm 3.5 MB/s | 3.7 MB 00:01 2023-12-06T15:30:25.195 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:30:25.195 INFO:teuthology.orchestra.run.smithi204.stdout:Total 3.5 MB/s | 3.7 MB 00:01 2023-12-06T15:30:25.195 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:30:25.207 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:30:25.207 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:30:25.270 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:30:25.273 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:30:25.709 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:30:25.764 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:48 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:25.918 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:26.333 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:27.020 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:27.020 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:27.021 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:30:27.021 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:30:27.021 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:27.021 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:30:27.215 INFO:teuthology.orchestra.run.smithi176.stdout:Package librados2-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:27.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-nbd 2023-12-06T15:30:27.440 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:27.441 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-06T15:30:27.441 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:27.684 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librbd1 2023-12-06T15:30:28.090 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:03:00 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:30:28.539 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:51 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:29.765 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:30:29.766 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:30:29.766 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2023-12-06T15:30:29.766 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:30:29.766 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2023-12-06T15:30:29.766 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 175 k 2023-12-06T15:30:29.766 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:29.766 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2023-12-06T15:30:29.766 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2023-12-06T15:30:29.766 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2023-12-06T15:30:29.766 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:29.767 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 175 k 2023-12-06T15:30:29.767 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 523 k 2023-12-06T15:30:29.767 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2023-12-06T15:30:30.000 INFO:teuthology.orchestra.run.smithi176.stdout:Package librbd1-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:30.048 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-nbd-16.2.14-335.gd03fca4f.el8.x86_64.rpm 626 kB/s | 175 kB 00:00 2023-12-06T15:30:30.048 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:30:30.048 INFO:teuthology.orchestra.run.smithi204.stdout:Total 622 kB/s | 175 kB 00:00 2023-12-06T15:30:30.049 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2023-12-06T15:30:30.059 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2023-12-06T15:30:30.059 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2023-12-06T15:30:30.108 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2023-12-06T15:30:30.111 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2023-12-06T15:30:30.222 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2023-12-06T15:30:30.226 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:30.227 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-06T15:30:30.227 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:30.332 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rados 2023-12-06T15:30:30.417 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-nbd-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:31.183 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:54 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:31.365 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:31.365 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:31.365 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2023-12-06T15:30:31.365 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:30:31.365 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:30:31.366 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:30:31.587 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-mirror 2023-12-06T15:30:32.453 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:03:04 ago on Wed 06 Dec 2023 03:27:28 PM UTC. 2023-12-06T15:30:32.662 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rados-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:32.887 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:32.888 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-06T15:30:32.888 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:32.976 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rgw 2023-12-06T15:30:33.836 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:56 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:33.896 INFO:teuthology.orchestra.run.smithi204.stdout:Package rbd-mirror-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:34.120 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2023-12-06T15:30:34.121 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2023-12-06T15:30:34.121 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2023-12-06T15:30:34.230 DEBUG:teuthology.parallel:result is None 2023-12-06T15:30:35.304 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rgw-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:35.532 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:35.533 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-06T15:30:35.533 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:35.644 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-cephfs 2023-12-06T15:30:36.499 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:59 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:37.951 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-cephfs-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:38.179 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:38.180 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-06T15:30:38.180 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:38.281 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rbd 2023-12-06T15:30:39.144 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:02 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:40.589 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rbd-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:40.811 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:40.812 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-06T15:30:40.812 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:40.915 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-fuse 2023-12-06T15:30:41.770 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:04 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:43.460 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:43.460 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:43.460 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2023-12-06T15:30:43.460 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:43.460 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:30:43.460 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 88 k 2023-12-06T15:30:43.460 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:43.461 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:30:43.461 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:43.461 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-06T15:30:43.461 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:43.461 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 88 k 2023-12-06T15:30:43.461 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 240 k 2023-12-06T15:30:43.461 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:30:43.736 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-fuse-16.2.14-335.gd03fca4f.el8.x86_64.rpm 322 kB/s | 88 kB 00:00 2023-12-06T15:30:43.780 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:30:43.780 INFO:teuthology.orchestra.run.smithi176.stdout:Total 321 kB/s | 88 kB 00:00 2023-12-06T15:30:43.780 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:30:43.780 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:30:43.780 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:30:43.795 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:30:43.799 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:30:43.901 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:30:44.119 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:45.091 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:30:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:30:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:45.092 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:45.312 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-mirror 2023-12-06T15:30:46.178 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:09 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:47.839 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:47.839 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:47.839 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2023-12-06T15:30:47.839 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:47.839 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:30:47.840 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 3.7 M 2023-12-06T15:30:47.840 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:47.840 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:30:47.840 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:47.840 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-06T15:30:47.840 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:47.840 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 3.7 M 2023-12-06T15:30:47.840 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 15 M 2023-12-06T15:30:47.840 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:30:49.314 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-mirror-16.2.14-335.gd03fca4f.el8.x86_64.rpm 2.5 MB/s | 3.7 MB 00:01 2023-12-06T15:30:49.315 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:30:49.315 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.5 MB/s | 3.7 MB 00:01 2023-12-06T15:30:49.316 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:30:49.329 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:30:49.329 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:30:49.396 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:30:49.399 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:30:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:30:50.053 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:50.482 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:51.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:51.184 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:51.184 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:30:51.184 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:30:51.184 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:51.184 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:51.414 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-nbd 2023-12-06T15:30:52.282 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:15 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:53.974 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:53.975 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:53.975 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2023-12-06T15:30:53.975 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:53.975 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-06T15:30:53.975 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd x86_64 2:16.2.14-335.gd03fca4f.el8 ceph 175 k 2023-12-06T15:30:53.975 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:53.975 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-06T15:30:53.975 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-06T15:30:53.976 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-06T15:30:53.976 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:53.976 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 175 k 2023-12-06T15:30:53.976 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 523 k 2023-12-06T15:30:53.976 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-06T15:30:54.167 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-nbd-16.2.14-335.gd03fca4f.el8.x86_64.rpm 920 kB/s | 175 kB 00:00 2023-12-06T15:30:54.167 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-06T15:30:54.168 INFO:teuthology.orchestra.run.smithi176.stdout:Total 914 kB/s | 175 kB 00:00 2023-12-06T15:30:54.168 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-06T15:30:54.178 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-06T15:30:54.178 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-06T15:30:54.228 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-06T15:30:54.231 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-06T15:30:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-06T15:30:54.568 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:54.848 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-nbd-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:55.554 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-2:16.2.14-335.gd03fca4f.el8.x86_64 1/1 2023-12-06T15:30:55.554 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:55.554 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-06T15:30:55.555 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-2:16.2.14-335.gd03fca4f.el8.x86_64 2023-12-06T15:30:55.555 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:30:55.555 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:55.757 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-mirror 2023-12-06T15:30:56.619 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:19 ago on Wed 06 Dec 2023 03:27:37 PM UTC. 2023-12-06T15:30:58.076 INFO:teuthology.orchestra.run.smithi176.stdout:Package rbd-mirror-2:16.2.14-335.gd03fca4f.el8.x86_64 is already installed. 2023-12-06T15:30:58.299 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-06T15:30:58.300 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-06T15:30:58.300 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-06T15:30:58.385 DEBUG:teuthology.parallel:result is None 2023-12-06T15:30:58.385 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T15:30:58.538 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-06T15:30:58.560 INFO:teuthology.orchestra.run.smithi176.stdout:16.2.14-335.gd03fca4f.el8 2023-12-06T15:30:58.561 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-335.gd03fca4f.el8 2023-12-06T15:30:58.561 INFO:teuthology.task.install:The correct ceph version 16.2.14-335.gd03fca4f is installed. 2023-12-06T15:30:58.561 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T15:30:58.708 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-06T15:30:58.735 INFO:teuthology.orchestra.run.smithi204.stdout:16.2.14-335.gd03fca4f.el8 2023-12-06T15:30:58.735 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-335.gd03fca4f.el8 2023-12-06T15:30:58.735 INFO:teuthology.task.install:The correct ceph version 16.2.14-335.gd03fca4f is installed. 2023-12-06T15:30:58.735 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-06T15:30:58.736 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:30:58.736 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-06T15:30:58.762 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:30:58.762 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-06T15:30:58.807 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-06T15:30:58.807 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:30:58.807 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2023-12-06T15:30:58.833 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-06T15:30:58.900 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:30:58.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2023-12-06T15:30:58.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-06T15:30:59.000 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-06T15:30:59.000 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:30:59.000 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-06T15:30:59.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-06T15:30:59.094 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:30:59.094 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-06T15:30:59.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-06T15:30:59.193 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-06T15:30:59.193 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:30:59.193 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2023-12-06T15:30:59.220 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-06T15:30:59.287 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:30:59.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2023-12-06T15:30:59.317 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-06T15:30:59.386 INFO:teuthology.run_tasks:Running task ceph... 2023-12-06T15:30:59.449 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-06T15:30:59.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /var/log/ceph 2023-12-06T15:30:59.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /var/log/ceph 2023-12-06T15:30:59.480 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-06T15:30:59.480 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-06T15:30:59.519 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-06T15:30:59.548 INFO:tasks.ceph:Creating extra log directories... 2023-12-06T15:30:59.549 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-06T15:30:59.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-06T15:30:59.632 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-12-06T15:30:59.632 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-06T15:30:59.632 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-06_15:04:09-rbd-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7480288', 'branch': 'wip-yuri6-testing-2023-12-05-0753-pacific', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': None, 'first_in_suite': False, 'job_id': '7480288', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-12-06_15:04:09-rbd-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2023-12-05-0753-pacific'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2023-12-05-0753-pacific', 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5062, 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri6-testing-2023-12-05-0753-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b', 'targets': {'smithi176.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdeHQEar7hPqc96kDyNSh6M7LB7BwlUWoPVhBK4v+HkXWC8SO3dk4cEx/FQYQnJlhdNSE30SESLYGC+N2hMAnc=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD4ib3EayXX/EbckMYpTCjmhZ2CmtXTzz/JTEtV7vSVUI06kQZE5WQuceaANryCFJip34iVZ39qXWIo6HE17Fmo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-06_15:04:09', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-06T15:30:59.633 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-06T15:30:59.652 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-06T15:30:59.676 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2023-12-06T15:30:59.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2023-12-06T15:30:59.743 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:30:59.743 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2023-12-06T15:30:59.792 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-06T15:30:59.792 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2023-12-06T15:30:59.849 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-06T15:30:59.850 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T15:30:59.850 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 388915 Links: 1 2023-12-06T15:30:59.850 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T15:30:59.850 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T15:30:59.850 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2023-12-06 15:30:50.235399713 +0000 2023-12-06T15:30:59.850 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2023-12-06 15:26:15.975048689 +0000 2023-12-06T15:30:59.850 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2023-12-06 15:26:15.975048689 +0000 2023-12-06T15:30:59.850 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2023-12-06T15:30:59.850 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-06T15:30:59.914 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2023-12-06T15:30:59.915 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2023-12-06T15:30:59.915 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000195989 s, 2.6 MB/s 2023-12-06T15:30:59.916 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-06T15:30:59.973 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2023-12-06T15:31:00.031 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-06T15:31:00.031 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T15:31:00.031 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 396445 Links: 1 2023-12-06T15:31:00.031 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T15:31:00.031 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T15:31:00.031 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2023-12-06 15:30:50.235399713 +0000 2023-12-06T15:31:00.031 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2023-12-06 15:26:16.489043998 +0000 2023-12-06T15:31:00.031 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2023-12-06 15:26:16.489043998 +0000 2023-12-06T15:31:00.031 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2023-12-06T15:31:00.031 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-06T15:31:00.097 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2023-12-06T15:31:00.097 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2023-12-06T15:31:00.097 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000215938 s, 2.4 MB/s 2023-12-06T15:31:00.098 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-06T15:31:00.157 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2023-12-06T15:31:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-06T15:31:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T15:31:00.215 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 390669 Links: 1 2023-12-06T15:31:00.215 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T15:31:00.215 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T15:31:00.215 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2023-12-06 15:30:50.235399713 +0000 2023-12-06T15:31:00.216 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2023-12-06 15:26:17.002039316 +0000 2023-12-06T15:31:00.216 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2023-12-06 15:26:17.002039316 +0000 2023-12-06T15:31:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2023-12-06T15:31:00.216 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-06T15:31:00.280 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2023-12-06T15:31:00.280 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2023-12-06T15:31:00.280 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000187988 s, 2.7 MB/s 2023-12-06T15:31:00.281 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-06T15:31:00.340 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2023-12-06T15:31:00.398 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-06T15:31:00.398 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T15:31:00.398 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 6h/6d Inode: 395742 Links: 1 2023-12-06T15:31:00.398 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T15:31:00.399 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T15:31:00.399 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2023-12-06 15:30:50.235399713 +0000 2023-12-06T15:31:00.399 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2023-12-06 15:26:17.534034461 +0000 2023-12-06T15:31:00.399 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2023-12-06 15:26:17.534034461 +0000 2023-12-06T15:31:00.399 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2023-12-06T15:31:00.399 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-06T15:31:00.464 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2023-12-06T15:31:00.464 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2023-12-06T15:31:00.464 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000235629 s, 2.2 MB/s 2023-12-06T15:31:00.465 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-06T15:31:00.523 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-06T15:31:00.524 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-06T15:31:00.524 INFO:tasks.ceph:Generating config... 2023-12-06T15:31:00.526 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-06T15:31:00.526 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-06T15:31:00.526 INFO:tasks.ceph:[client] log to stderr = False 2023-12-06T15:31:00.526 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-06T15:31:00.526 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2023-12-06T15:31:00.526 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:00.526 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:00.526 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:00.526 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:00.526 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:00.526 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:00.526 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:00.527 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:00.527 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-06T15:31:00.527 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-06T15:31:00.527 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-06T15:31:00.527 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-06T15:31:00.527 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-06T15:31:00.527 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-06T15:31:00.527 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-06T15:31:00.527 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-06T15:31:00.527 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-06T15:31:00.527 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-06T15:31:00.528 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-06T15:31:00.528 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-06T15:31:00.528 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-06T15:31:00.528 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-06T15:31:00.528 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-06T15:31:00.528 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-06T15:31:00.528 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-06T15:31:00.528 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-06T15:31:00.528 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-06T15:31:00.528 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-06T15:31:00.528 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-06T15:31:00.528 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-06T15:31:00.529 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-12-06T15:31:00.529 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-12-06T15:31:00.674 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster1.keyring 2023-12-06T15:31:00.676 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-12-06T15:31:00.717 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-06T15:31:00.784 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.176')] 2023-12-06T15:31:00.784 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.176', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-06T15:31:00.785 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:00.785 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-06T15:31:00.842 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.176 --print /home/ubuntu/cephtest/cluster1.monmap 2023-12-06T15:31:00.920 INFO:teuthology.orchestra.run.smithi176.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-12-06T15:31:00.920 INFO:teuthology.orchestra.run.smithi176.stdout:monmaptool: generated fsid 04ea9659-2028-4895-8ed7-dcd087520c51 2023-12-06T15:31:00.920 INFO:teuthology.orchestra.run.smithi176.stdout:epoch 0 2023-12-06T15:31:00.920 INFO:teuthology.orchestra.run.smithi176.stdout:fsid 04ea9659-2028-4895-8ed7-dcd087520c51 2023-12-06T15:31:00.920 INFO:teuthology.orchestra.run.smithi176.stdout:last_changed 2023-12-06T15:31:00.920365+0000 2023-12-06T15:31:00.920 INFO:teuthology.orchestra.run.smithi176.stdout:created 2023-12-06T15:31:00.920365+0000 2023-12-06T15:31:00.920 INFO:teuthology.orchestra.run.smithi176.stdout:min_mon_release 0 (unknown) 2023-12-06T15:31:00.920 INFO:teuthology.orchestra.run.smithi176.stdout:election_strategy: 1 2023-12-06T15:31:00.920 INFO:teuthology.orchestra.run.smithi176.stdout:0: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.a 2023-12-06T15:31:00.920 INFO:teuthology.orchestra.run.smithi176.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-12-06T15:31:00.921 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-06T15:31:00.979 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 04ea9659-2028-4895-8ed7-dcd087520c51... 2023-12-06T15:31:00.980 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-06T15:31:01.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-06T15:31:01.063 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2023-12-06T15:31:01.063 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2023-12-06T15:31:01.063 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-06T15:31:01.063 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-06T15:31:01.064 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2023-12-06T15:31:01.065 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.065 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2023-12-06T15:31:01.065 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-06T15:31:01.065 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.065 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2023-12-06T15:31:01.065 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.065 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2023-12-06T15:31:01.065 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2023-12-06T15:31:01.065 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2023-12-06T15:31:01.065 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2023-12-06T15:31:01.065 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2023-12-06T15:31:01.065 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: mon_allow_pool_size_one = true 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2023-12-06T15:31:01.066 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2023-12-06T15:31:01.067 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2023-12-06T15:31:01.067 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = 172.21.15.176 2023-12-06T15:31:01.067 INFO:teuthology.orchestra.run.smithi176.stdout: mon client directed command retry = 5 2023-12-06T15:31:01.067 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject socket failures = 5000 2023-12-06T15:31:01.067 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = 04ea9659-2028-4895-8ed7-dcd087520c51 2023-12-06T15:31:01.067 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.067 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2023-12-06T15:31:01.067 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2023-12-06T15:31:01.067 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.067 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2023-12-06T15:31:01.067 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2023-12-06T15:31:01.067 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.068 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2023-12-06T15:31:01.069 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.069 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2023-12-06T15:31:01.069 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.069 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2023-12-06T15:31:01.069 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2023-12-06T15:31:01.069 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.069 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2023-12-06T15:31:01.069 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2023-12-06T15:31:01.069 INFO:teuthology.orchestra.run.smithi176.stdout: bdev async discard = True 2023-12-06T15:31:01.069 INFO:teuthology.orchestra.run.smithi176.stdout: bdev enable discard = True 2023-12-06T15:31:01.069 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore allocator = stupid 2023-12-06T15:31:01.069 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2023-12-06T15:31:01.070 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2023-12-06T15:31:01.070 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 1/20 2023-12-06T15:31:01.070 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 1/20 2023-12-06T15:31:01.070 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2023-12-06T15:31:01.070 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2023-12-06T15:31:01.070 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 4/10 2023-12-06T15:31:01.070 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2023-12-06T15:31:01.070 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2023-12-06T15:31:01.070 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2023-12-06T15:31:01.070 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2023-12-06T15:31:01.070 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2023-12-06T15:31:01.070 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2023-12-06T15:31:01.071 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.072 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: debug rbd = 20 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: debug rbd_mirror = 15 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: log to stderr = False 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: rbd default features = 125 2023-12-06T15:31:01.073 INFO:teuthology.orchestra.run.smithi176.stdout: rbd mirror image policy type = none 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.0] 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.1] 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.2] 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.3] 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:01.074 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:01.075 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2023-12-06T15:31:01.075 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2023-12-06T15:31:01.075 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-06T15:31:01.075 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-06T15:31:01.076 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2023-12-06T15:31:01.077 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-06T15:31:01.078 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-06T15:31:01.078 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2023-12-06T15:31:01.078 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.078 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-06T15:31:01.078 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.078 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2023-12-06T15:31:01.078 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.078 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2023-12-06T15:31:01.078 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.078 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2023-12-06T15:31:01.078 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2023-12-06T15:31:01.078 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = 172.21.15.176 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: mon client directed command retry = 5 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject socket failures = 5000 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = 04ea9659-2028-4895-8ed7-dcd087520c51 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2023-12-06T15:31:01.079 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2023-12-06T15:31:01.080 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: bdev async discard = True 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: bdev enable discard = True 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore allocator = stupid 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore block size = 96636764160 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore fsck on mount = True 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluefs = 1/20 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluestore = 1/20 2023-12-06T15:31:01.081 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout: debug rocksdb = 4/10 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd backfillfull_ratio = 0.85 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd full ratio = 0.9 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd nearfull ratio = 0.8 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout: osd failsafe full ratio = 0.95 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout: osd objectstore = bluestore 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2023-12-06T15:31:01.082 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2023-12-06T15:31:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2023-12-06T15:31:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2023-12-06T15:31:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-06T15:31:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2023-12-06T15:31:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2023-12-06T15:31:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2023-12-06T15:31:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-06T15:31:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-06T15:31:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:01.084 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2023-12-06T15:31:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: debug rbd = 20 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: debug rbd_mirror = 15 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: log to stderr = False 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: rbd default features = 125 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: rbd mirror image policy type = none 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.0] 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:01.085 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.1] 2023-12-06T15:31:01.086 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:01.086 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:01.086 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.2] 2023-12-06T15:31:01.086 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:01.086 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:01.086 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.3] 2023-12-06T15:31:01.086 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:01.086 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:01.087 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-12-06T15:31:01.088 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-12-06T15:31:01.180 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-06T15:31:01.181 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:01.181 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-12-06T15:31:01.198 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:01.199 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-12-06T15:31:01.255 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2023-12-06T15:31:01.255 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:01.256 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-06T15:31:01.256 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-06T15:31:01.332 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:01.332 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-06T15:31:01.391 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2023-12-06T15:31:01.391 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:01.391 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-06T15:31:01.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-06T15:31:01.437 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:01.437 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-06T15:31:01.497 INFO:tasks.ceph:Setting up mon nodes... 2023-12-06T15:31:01.497 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-06T15:31:01.497 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-06T15:31:01.546 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-06T15:31:01.549 INFO:tasks.ceph:Setting up mds nodes... 2023-12-06T15:31:01.549 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-06T15:31:01.549 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-12-06T15:31:01.588 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-12-06T15:31:01.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-12-06T15:31:01.725 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-12-06T15:31:01.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-06T15:31:01.779 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-06T15:31:01.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-06T15:31:01.871 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-06T15:31:01.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-06T15:31:01.964 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-06T15:31:01.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-06T15:31:02.055 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-06T15:31:02.067 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-06T15:31:02.146 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-06T15:31:02.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-06T15:31:02.237 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-06T15:31:02.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-06T15:31:02.328 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-06T15:31:02.340 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-06T15:31:02.341 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-06T15:31:02.341 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-12-06T15:31:02.371 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-06T15:31:02.371 INFO:tasks.ceph:role: cluster1.osd.0 2023-12-06T15:31:02.371 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com 2023-12-06T15:31:02.372 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-06T15:31:02.448 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T15:31:02.449 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T15:31:02.449 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T15:31:02.449 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-06T15:31:02.449 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T15:31:02.449 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2023-12-06T15:31:02.449 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T15:31:02.449 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-06T15:31:02.449 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T15:31:02.449 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T15:31:02.880 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2023-12-06T15:31:02.882 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2023-12-06T15:31:02.882 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-12-06T15:31:03.063 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-12-06T15:31:03.103 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-12-06T15:31:03.172 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-06T15:31:03.172 INFO:tasks.ceph:role: cluster1.osd.1 2023-12-06T15:31:03.172 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com 2023-12-06T15:31:03.173 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-06T15:31:03.248 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T15:31:03.248 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T15:31:03.248 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T15:31:03.248 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-06T15:31:03.249 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T15:31:03.249 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2023-12-06T15:31:03.249 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T15:31:03.249 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-06T15:31:03.249 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T15:31:03.249 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T15:31:03.668 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2023-12-06T15:31:03.670 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2023-12-06T15:31:03.670 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-12-06T15:31:03.782 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-12-06T15:31:03.816 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-12-06T15:31:03.885 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-06T15:31:03.885 INFO:tasks.ceph:role: cluster1.osd.2 2023-12-06T15:31:03.885 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com 2023-12-06T15:31:03.886 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-06T15:31:03.961 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T15:31:03.961 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T15:31:03.961 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T15:31:03.962 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-06T15:31:03.962 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T15:31:03.962 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2023-12-06T15:31:03.962 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T15:31:03.962 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-06T15:31:03.962 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T15:31:03.962 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T15:31:04.408 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2023-12-06T15:31:04.410 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2023-12-06T15:31:04.410 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-12-06T15:31:04.571 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-12-06T15:31:04.607 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-06T15:31:04.703 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:04.702+0000 7f2c5c93a380 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-12-06T15:31:04.703 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:04.703+0000 7f2c5c93a380 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-12-06T15:31:04.703 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:04.703+0000 7f2c5c93a380 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-06T15:31:04.704 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:04.703+0000 7f2c5c93a380 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-06T15:31:04.704 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:04.703+0000 7f2c5c93a380 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-06T15:31:04.711 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:04.710+0000 7f2c5c93a380 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-12-06T15:31:07.552 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-12-06T15:31:07.584 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-06T15:31:07.674 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:07.673+0000 7efc5515f380 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-12-06T15:31:07.674 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:07.673+0000 7efc5515f380 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-12-06T15:31:07.675 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:07.674+0000 7efc5515f380 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-06T15:31:07.675 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:07.674+0000 7efc5515f380 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-06T15:31:07.675 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:07.674+0000 7efc5515f380 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-06T15:31:07.675 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:07.674+0000 7efc5515f380 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-12-06T15:31:10.517 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-12-06T15:31:10.546 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-06T15:31:10.633 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:10.632+0000 7fd2d8a1a380 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-12-06T15:31:10.633 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:10.632+0000 7fd2d8a1a380 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-12-06T15:31:10.633 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:10.632+0000 7fd2d8a1a380 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-06T15:31:10.633 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:10.632+0000 7fd2d8a1a380 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-06T15:31:10.634 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:10.632+0000 7fd2d8a1a380 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-06T15:31:10.634 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-06T15:31:10.633+0000 7fd2d8a1a380 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-12-06T15:31:13.477 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-12-06T15:31:13.508 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-06T15:31:13.509 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:13.509 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-12-06T15:31:13.579 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:13.579 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-12-06T15:31:13.649 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:13.649 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-12-06T15:31:13.718 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:13.719 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-12-06T15:31:13.787 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:13.787 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-12-06T15:31:13.847 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:13.847 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-12-06T15:31:13.869 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:13.869 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-12-06T15:31:13.930 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:13.931 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-12-06T15:31:13.991 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:13.992 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-12-06T15:31:14.052 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:14.052 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-12-06T15:31:14.112 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:14.113 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-12-06T15:31:14.173 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:14.173 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-12-06T15:31:14.232 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:14.232 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-12-06T15:31:14.291 INFO:tasks.ceph:Adding keys to all mons... 2023-12-06T15:31:14.291 DEBUG:teuthology.orchestra.run.smithi176:> sudo tee -a /etc/ceph/cluster1.keyring 2023-12-06T15:31:14.322 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.x] 2023-12-06T15:31:14.322 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC1k3BlRsigIBAAF63N6gG5e0eWO7D4YyJ/+g== 2023-12-06T15:31:14.322 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.0] 2023-12-06T15:31:14.322 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC4k3Bl4L7uKRAAwzDqixmeVzos3ksJMVcuZQ== 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.1] 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC7k3Blv6Y3KBAA6ZAVH2fArOdoftYh1qSpLg== 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.2] 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC+k3BlJwC9JRAAZVf+nwqh1DeQpVikpOH7ew== 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout:[client.0] 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC1k3BlHUggIxAAyNx5bP7rKF40d4ev15bSYg== 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror] 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC1k3BlkgI+KxAAx62mCeMdQStddcqfFlJrjg== 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.0] 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC1k3BlAud2LhAAK5aqcylfYz0clsHQFFkKjw== 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.1] 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC1k3BlYWPyMxAA9Qqp7woNtdTtsp/symwOhQ== 2023-12-06T15:31:14.323 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.2] 2023-12-06T15:31:14.324 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC1k3Bl9eh2ORAAgpJ7rsHurbOCkRJO7YL+3A== 2023-12-06T15:31:14.324 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.3] 2023-12-06T15:31:14.324 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC2k3Bl5FNXAxAAWDiDcE0RXVOJFeYpg9IkGA== 2023-12-06T15:31:14.324 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.4] 2023-12-06T15:31:14.324 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC2k3BlqwbGCBAAXxht1SxXv0vywG5QWMz6lw== 2023-12-06T15:31:14.324 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.5] 2023-12-06T15:31:14.324 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC2k3Bl8JEwDhAAY9LhdS2TkNPgcSFFTcLfRQ== 2023-12-06T15:31:14.324 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.6] 2023-12-06T15:31:14.324 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC2k3BlJROjExAAGrjE2Pm1q9n3vow8iGSx9Q== 2023-12-06T15:31:14.325 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-06T15:31:14.420 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T15:31:14.508 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T15:31:14.557 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T15:31:14.606 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:14.657 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:14.708 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:14.759 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:14.811 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:14.861 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:14.912 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:14.963 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:15.017 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:15.068 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-06T15:31:15.069 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-12-06T15:31:15.095 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-12-06T15:31:15.189 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-12-06T15:31:15.216 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-12-06T15:31:15.273 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-12-06T15:31:15.273 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-12-06T15:31:15.273 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-12-06T15:31:15.315 INFO:tasks.ceph.cluster1.mon.a:Started 2023-12-06T15:31:15.316 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-12-06T15:31:15.316 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-12-06T15:31:15.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-12-06T15:31:15.318 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-12-06T15:31:15.318 INFO:tasks.ceph:Setting crush tunables to default 2023-12-06T15:31:15.318 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster cluster1 osd crush tunables default 2023-12-06T15:31:15.535 INFO:teuthology.orchestra.run.smithi176.stderr:adjusted tunables profile to default 2023-12-06T15:31:15.546 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-12-06T15:31:15.546 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:15.546 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-12-06T15:31:15.573 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:15.573 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-12-06T15:31:15.640 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:15.641 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-12-06T15:31:15.707 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster cluster1 osd new f3fd59e7-6c6a-4e34-9669-31e02e3747f7 0 2023-12-06T15:31:15.736 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:15.736+0000 7f7ba387b680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-06T15:31:15.809 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:15.808+0000 7f7ba387b680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-06T15:31:15.941 INFO:teuthology.orchestra.run.smithi176.stdout:0 2023-12-06T15:31:15.954 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster cluster1 osd new 80806cae-718a-42ba-b880-be7dfba83802 1 2023-12-06T15:31:16.162 INFO:teuthology.orchestra.run.smithi176.stdout:1 2023-12-06T15:31:16.174 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster cluster1 osd new d1d4ef0b-bbe7-493d-94c5-9f4507c35d23 2 2023-12-06T15:31:16.251 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:16.251+0000 7f7ba387b680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-06T15:31:16.380 INFO:teuthology.orchestra.run.smithi176.stdout:2 2023-12-06T15:31:16.392 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-12-06T15:31:16.393 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-12-06T15:31:16.395 INFO:tasks.ceph.cluster1.osd.0:Started 2023-12-06T15:31:16.395 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-12-06T15:31:16.395 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-12-06T15:31:16.397 INFO:tasks.ceph.cluster1.osd.1:Started 2023-12-06T15:31:16.398 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-12-06T15:31:16.398 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-12-06T15:31:16.400 INFO:tasks.ceph.cluster1.osd.2:Started 2023-12-06T15:31:16.400 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-06T15:31:16.598 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:16.599 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":5,"fsid":"04ea9659-2028-4895-8ed7-dcd087520c51","created":"2023-12-06T15:31:15.435621+0000","modified":"2023-12-06T15:31:16.379480+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T15:31:16.379480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"f3fd59e7-6c6a-4e34-9669-31e02e3747f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"80806cae-718a-42ba-b880-be7dfba83802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d1d4ef0b-bbe7-493d-94c5-9f4507c35d23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T15:31:16.611 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-12-06T15:31:16.611 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-06T15:31:16.913 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-06T15:31:17.002 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:31:17.002+0000 7f52426af380 -1 Falling back to public interface 2023-12-06T15:31:17.017 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:17.016+0000 7f7ba387b680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-06T15:31:17.020 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T15:31:17.020+0000 7f077c136380 -1 Falling back to public interface 2023-12-06T15:31:17.029 INFO:tasks.ceph.cluster1.osd.1.smithi176.stderr:2023-12-06T15:31:17.028+0000 7f5a1d10b380 -1 Falling back to public interface 2023-12-06T15:31:17.088 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:17.087+0000 7f7ba387b680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-06T15:31:17.277 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:17.276+0000 7f7ba387b680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-06T15:31:17.322 INFO:teuthology.misc.health.smithi176.stdout: 2023-12-06T15:31:17.322 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":5,"fsid":"04ea9659-2028-4895-8ed7-dcd087520c51","created":"2023-12-06T15:31:15.435621+0000","modified":"2023-12-06T15:31:16.379480+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T15:31:16.379480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"f3fd59e7-6c6a-4e34-9669-31e02e3747f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"80806cae-718a-42ba-b880-be7dfba83802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d1d4ef0b-bbe7-493d-94c5-9f4507c35d23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T15:31:17.334 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-06T15:31:17.717 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:17.717+0000 7f7ba387b680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-06T15:31:17.819 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:17.818+0000 7f7ba387b680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-06T15:31:17.974 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:17.974+0000 7f7ba387b680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-06T15:31:18.051 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:18.050+0000 7f7ba387b680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-06T15:31:18.119 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:18.119+0000 7f7ba387b680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-06T15:31:18.378 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:18.378+0000 7f7ba387b680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-06T15:31:18.579 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:18.579+0000 7f7ba387b680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-06T15:31:19.246 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:19.245+0000 7f7ba387b680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-06T15:31:19.315 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:19.315+0000 7f7ba387b680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-06T15:31:19.393 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:19.393+0000 7f7ba387b680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-06T15:31:19.552 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:19.551+0000 7f7ba387b680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-06T15:31:19.630 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:19.629+0000 7f7ba387b680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-06T15:31:19.747 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:19.746+0000 7f7ba387b680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-06T15:31:19.881 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:19.880+0000 7f7ba387b680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-06T15:31:20.209 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:20.208+0000 7f7ba387b680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-06T15:31:20.263 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:31:20.263+0000 7f52426af380 -1 osd.2 0 log_to_monitors {default=true} 2023-12-06T15:31:20.286 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:20.285+0000 7f7ba387b680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-06T15:31:20.541 INFO:tasks.ceph.cluster1.osd.1.smithi176.stderr:2023-12-06T15:31:20.540+0000 7f5a1d10b380 -1 osd.1 0 log_to_monitors {default=true} 2023-12-06T15:31:20.552 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T15:31:20.552+0000 7f077c136380 -1 osd.0 0 log_to_monitors {default=true} 2023-12-06T15:31:21.305 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T15:31:21.305+0000 7f7b58d94700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-06T15:31:21.326 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:31:21.326+0000 7f5239050700 -1 osd.2 0 waiting for initial osdmap 2023-12-06T15:31:21.330 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:31:21.330+0000 7f52357cc700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T15:31:22.328 INFO:tasks.ceph.cluster1.osd.1.smithi176.stderr:2023-12-06T15:31:22.327+0000 7f5a13aac700 -1 osd.1 0 waiting for initial osdmap 2023-12-06T15:31:22.328 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T15:31:22.327+0000 7f0772ad7700 -1 osd.0 0 waiting for initial osdmap 2023-12-06T15:31:22.333 INFO:tasks.ceph.cluster1.osd.1.smithi176.stderr:2023-12-06T15:31:22.332+0000 7f5a10228700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T15:31:22.333 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T15:31:22.333+0000 7f076f253700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T15:31:23.638 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-06T15:31:23.929 INFO:teuthology.misc.health.smithi176.stdout: 2023-12-06T15:31:23.929 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":9,"fsid":"04ea9659-2028-4895-8ed7-dcd087520c51","created":"2023-12-06T15:31:15.435621+0000","modified":"2023-12-06T15:31:23.334520+0000","last_up_change":"2023-12-06T15:31:23.334520+0000","last_in_change":"2023-12-06T15:31:16.379480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"f3fd59e7-6c6a-4e34-9669-31e02e3747f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":114688},{"type":"v1","addr":"172.21.15.176:6809","nonce":114688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":114688},{"type":"v1","addr":"172.21.15.176:6811","nonce":114688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":114688},{"type":"v1","addr":"172.21.15.176:6815","nonce":114688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":114688},{"type":"v1","addr":"172.21.15.176:6813","nonce":114688}]},"public_addr":"172.21.15.176:6809/114688","cluster_addr":"172.21.15.176:6811/114688","heartbeat_back_addr":"172.21.15.176:6815/114688","heartbeat_front_addr":"172.21.15.176:6813/114688","state":["exists","up"]},{"osd":1,"uuid":"80806cae-718a-42ba-b880-be7dfba83802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":114695},{"type":"v1","addr":"172.21.15.176:6817","nonce":114695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":114695},{"type":"v1","addr":"172.21.15.176:6819","nonce":114695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":114695},{"type":"v1","addr":"172.21.15.176:6823","nonce":114695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":114695},{"type":"v1","addr":"172.21.15.176:6821","nonce":114695}]},"public_addr":"172.21.15.176:6817/114695","cluster_addr":"172.21.15.176:6819/114695","heartbeat_back_addr":"172.21.15.176:6823/114695","heartbeat_front_addr":"172.21.15.176:6821/114695","state":["exists","up"]},{"osd":2,"uuid":"d1d4ef0b-bbe7-493d-94c5-9f4507c35d23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":114685},{"type":"v1","addr":"172.21.15.176:6801","nonce":114685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":114685},{"type":"v1","addr":"172.21.15.176:6803","nonce":114685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":114685},{"type":"v1","addr":"172.21.15.176:6807","nonce":114685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":114685},{"type":"v1","addr":"172.21.15.176:6805","nonce":114685}]},"public_addr":"172.21.15.176:6801/114685","cluster_addr":"172.21.15.176:6803/114685","heartbeat_back_addr":"172.21.15.176:6807/114685","heartbeat_front_addr":"172.21.15.176:6805/114685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:21.234599+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T15:31:23.941 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-06T15:31:23.941 INFO:tasks.ceph:Creating RBD pool 2023-12-06T15:31:23.941 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-12-06T15:31:24.343 INFO:teuthology.orchestra.run.smithi176.stderr:pool 'rbd' created 2023-12-06T15:31:24.354 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-06T15:31:25.347 INFO:teuthology.orchestra.run.smithi176.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-06T15:31:25.362 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-12-06T15:31:25.362 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-12-06T15:31:25.362 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-12-06T15:31:25.363 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-12-06T15:31:25.363 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-06T15:31:25.697 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:25.709 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":114278},{"type":"v1","addr":"172.21.15.176:6825","nonce":114278}]},"active_addr":"172.21.15.176:6825/114278","active_change":"2023-12-06T15:31:20.300072+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":3916708084}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":3222119229}]}]} 2023-12-06T15:31:25.711 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-12-06T15:31:25.711 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-12-06T15:31:25.711 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-06T15:31:25.997 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:25.997 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":11,"fsid":"04ea9659-2028-4895-8ed7-dcd087520c51","created":"2023-12-06T15:31:15.435621+0000","modified":"2023-12-06T15:31:25.337770+0000","last_up_change":"2023-12-06T15:31:23.334520+0000","last_in_change":"2023-12-06T15:31:16.379480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-06T15:31:23.370481+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T15:31:24.258167+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f3fd59e7-6c6a-4e34-9669-31e02e3747f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":114688},{"type":"v1","addr":"172.21.15.176:6809","nonce":114688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":114688},{"type":"v1","addr":"172.21.15.176:6811","nonce":114688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":114688},{"type":"v1","addr":"172.21.15.176:6815","nonce":114688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":114688},{"type":"v1","addr":"172.21.15.176:6813","nonce":114688}]},"public_addr":"172.21.15.176:6809/114688","cluster_addr":"172.21.15.176:6811/114688","heartbeat_back_addr":"172.21.15.176:6815/114688","heartbeat_front_addr":"172.21.15.176:6813/114688","state":["exists","up"]},{"osd":1,"uuid":"80806cae-718a-42ba-b880-be7dfba83802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":114695},{"type":"v1","addr":"172.21.15.176:6817","nonce":114695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":114695},{"type":"v1","addr":"172.21.15.176:6819","nonce":114695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":114695},{"type":"v1","addr":"172.21.15.176:6823","nonce":114695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":114695},{"type":"v1","addr":"172.21.15.176:6821","nonce":114695}]},"public_addr":"172.21.15.176:6817/114695","cluster_addr":"172.21.15.176:6819/114695","heartbeat_back_addr":"172.21.15.176:6823/114695","heartbeat_front_addr":"172.21.15.176:6821/114695","state":["exists","up"]},{"osd":2,"uuid":"d1d4ef0b-bbe7-493d-94c5-9f4507c35d23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":114685},{"type":"v1","addr":"172.21.15.176:6801","nonce":114685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":114685},{"type":"v1","addr":"172.21.15.176:6803","nonce":114685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":114685},{"type":"v1","addr":"172.21.15.176:6807","nonce":114685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":114685},{"type":"v1","addr":"172.21.15.176:6805","nonce":114685}]},"public_addr":"172.21.15.176:6801/114685","cluster_addr":"172.21.15.176:6803/114685","heartbeat_back_addr":"172.21.15.176:6807/114685","heartbeat_front_addr":"172.21.15.176:6805/114685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:21.566125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:21.526441+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:21.234599+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T15:31:26.011 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-12-06T15:31:26.011 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-06T15:31:26.296 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:26.297 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":11,"fsid":"04ea9659-2028-4895-8ed7-dcd087520c51","created":"2023-12-06T15:31:15.435621+0000","modified":"2023-12-06T15:31:25.337770+0000","last_up_change":"2023-12-06T15:31:23.334520+0000","last_in_change":"2023-12-06T15:31:16.379480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-06T15:31:23.370481+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T15:31:24.258167+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f3fd59e7-6c6a-4e34-9669-31e02e3747f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":114688},{"type":"v1","addr":"172.21.15.176:6809","nonce":114688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":114688},{"type":"v1","addr":"172.21.15.176:6811","nonce":114688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":114688},{"type":"v1","addr":"172.21.15.176:6815","nonce":114688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":114688},{"type":"v1","addr":"172.21.15.176:6813","nonce":114688}]},"public_addr":"172.21.15.176:6809/114688","cluster_addr":"172.21.15.176:6811/114688","heartbeat_back_addr":"172.21.15.176:6815/114688","heartbeat_front_addr":"172.21.15.176:6813/114688","state":["exists","up"]},{"osd":1,"uuid":"80806cae-718a-42ba-b880-be7dfba83802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":114695},{"type":"v1","addr":"172.21.15.176:6817","nonce":114695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":114695},{"type":"v1","addr":"172.21.15.176:6819","nonce":114695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":114695},{"type":"v1","addr":"172.21.15.176:6823","nonce":114695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":114695},{"type":"v1","addr":"172.21.15.176:6821","nonce":114695}]},"public_addr":"172.21.15.176:6817/114695","cluster_addr":"172.21.15.176:6819/114695","heartbeat_back_addr":"172.21.15.176:6823/114695","heartbeat_front_addr":"172.21.15.176:6821/114695","state":["exists","up"]},{"osd":2,"uuid":"d1d4ef0b-bbe7-493d-94c5-9f4507c35d23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":114685},{"type":"v1","addr":"172.21.15.176:6801","nonce":114685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":114685},{"type":"v1","addr":"172.21.15.176:6803","nonce":114685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":114685},{"type":"v1","addr":"172.21.15.176:6807","nonce":114685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":114685},{"type":"v1","addr":"172.21.15.176:6805","nonce":114685}]},"public_addr":"172.21.15.176:6801/114685","cluster_addr":"172.21.15.176:6803/114685","heartbeat_back_addr":"172.21.15.176:6807/114685","heartbeat_front_addr":"172.21.15.176:6805/114685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:21.566125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:21.526441+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:21.234599+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T15:31:26.310 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-12-06T15:31:26.460 INFO:teuthology.orchestra.run.smithi176.stdout:38654705667 2023-12-06T15:31:26.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-12-06T15:31:26.608 INFO:teuthology.orchestra.run.smithi176.stdout:38654705667 2023-12-06T15:31:26.608 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-12-06T15:31:26.755 INFO:teuthology.orchestra.run.smithi176.stdout:34359738371 2023-12-06T15:31:26.755 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-06T15:31:27.039 INFO:teuthology.orchestra.run.smithi176.stdout:38654705666 2023-12-06T15:31:27.053 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705667 got 38654705666 for osd.0 2023-12-06T15:31:28.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-06T15:31:28.354 INFO:teuthology.orchestra.run.smithi176.stdout:38654705667 2023-12-06T15:31:28.367 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705667 got 38654705667 for osd.0 2023-12-06T15:31:28.367 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-06T15:31:28.656 INFO:teuthology.orchestra.run.smithi176.stdout:38654705667 2023-12-06T15:31:28.669 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705667 got 38654705667 for osd.1 2023-12-06T15:31:28.669 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-06T15:31:28.956 INFO:teuthology.orchestra.run.smithi176.stdout:34359738372 2023-12-06T15:31:28.970 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738372 for osd.2 2023-12-06T15:31:28.970 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-06T15:31:28.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-06T15:31:29.254 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:29.256 INFO:teuthology.orchestra.run.smithi176.stderr:dumped all 2023-12-06T15:31:29.267 INFO:teuthology.orchestra.run.smithi176.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-12-06T15:31:28.304502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891580,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223940,"statfs":{"total":289910292480,"available":288997314560,"internally_reserved":0,"allocated":299008,"data_stored":78601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.361264+0000","last_change":"2023-12-06T15:31:25.361264+0000","last_active":"2023-12-06T15:31:25.361264+0000","last_peered":"2023-12-06T15:31:25.361264+0000","last_clean":"2023-12-06T15:31:25.361264+0000","last_became_active":"2023-12-06T15:31:25.360985+0000","last_became_peered":"2023-12-06T15:31:25.360985+0000","last_unstale":"2023-12-06T15:31:25.361264+0000","last_undegraded":"2023-12-06T15:31:25.361264+0000","last_fullsized":"2023-12-06T15:31:25.361264+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.360417+0000","last_change":"2023-12-06T15:31:25.360417+0000","last_active":"2023-12-06T15:31:25.360417+0000","last_peered":"2023-12-06T15:31:25.360417+0000","last_clean":"2023-12-06T15:31:25.360417+0000","last_became_active":"2023-12-06T15:31:25.360097+0000","last_became_peered":"2023-12-06T15:31:25.360097+0000","last_unstale":"2023-12-06T15:31:25.360417+0000","last_undegraded":"2023-12-06T15:31:25.360417+0000","last_fullsized":"2023-12-06T15:31:25.360417+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.360590+0000","last_change":"2023-12-06T15:31:25.360590+0000","last_active":"2023-12-06T15:31:25.360590+0000","last_peered":"2023-12-06T15:31:25.360590+0000","last_clean":"2023-12-06T15:31:25.360590+0000","last_became_active":"2023-12-06T15:31:25.360289+0000","last_became_peered":"2023-12-06T15:31:25.360289+0000","last_unstale":"2023-12-06T15:31:25.360590+0000","last_undegraded":"2023-12-06T15:31:25.360590+0000","last_fullsized":"2023-12-06T15:31:25.360590+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.350181+0000","last_change":"2023-12-06T15:31:25.346677+0000","last_active":"2023-12-06T15:31:25.350181+0000","last_peered":"2023-12-06T15:31:25.350181+0000","last_clean":"2023-12-06T15:31:25.350181+0000","last_became_active":"2023-12-06T15:31:25.346360+0000","last_became_peered":"2023-12-06T15:31:25.346360+0000","last_unstale":"2023-12-06T15:31:25.350181+0000","last_undegraded":"2023-12-06T15:31:25.350181+0000","last_fullsized":"2023-12-06T15:31:25.350181+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.360336+0000","last_change":"2023-12-06T15:31:25.360336+0000","last_active":"2023-12-06T15:31:25.360336+0000","last_peered":"2023-12-06T15:31:25.360336+0000","last_clean":"2023-12-06T15:31:25.360336+0000","last_became_active":"2023-12-06T15:31:25.360016+0000","last_became_peered":"2023-12-06T15:31:25.360016+0000","last_unstale":"2023-12-06T15:31:25.360336+0000","last_undegraded":"2023-12-06T15:31:25.360336+0000","last_fullsized":"2023-12-06T15:31:25.360336+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.345634+0000","last_change":"2023-12-06T15:31:25.345634+0000","last_active":"2023-12-06T15:31:25.345634+0000","last_peered":"2023-12-06T15:31:25.345634+0000","last_clean":"2023-12-06T15:31:25.345634+0000","last_became_active":"2023-12-06T15:31:25.345340+0000","last_became_peered":"2023-12-06T15:31:25.345340+0000","last_unstale":"2023-12-06T15:31:25.345634+0000","last_undegraded":"2023-12-06T15:31:25.345634+0000","last_fullsized":"2023-12-06T15:31:25.345634+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.353041+0000","last_change":"2023-12-06T15:31:25.348740+0000","last_active":"2023-12-06T15:31:25.353041+0000","last_peered":"2023-12-06T15:31:25.353041+0000","last_clean":"2023-12-06T15:31:25.353041+0000","last_became_active":"2023-12-06T15:31:25.348397+0000","last_became_peered":"2023-12-06T15:31:25.348397+0000","last_unstale":"2023-12-06T15:31:25.353041+0000","last_undegraded":"2023-12-06T15:31:25.353041+0000","last_fullsized":"2023-12-06T15:31:25.353041+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.359996+0000","last_change":"2023-12-06T15:31:25.359996+0000","last_active":"2023-12-06T15:31:25.359996+0000","last_peered":"2023-12-06T15:31:25.359996+0000","last_clean":"2023-12-06T15:31:25.359996+0000","last_became_active":"2023-12-06T15:31:25.359616+0000","last_became_peered":"2023-12-06T15:31:25.359616+0000","last_unstale":"2023-12-06T15:31:25.359996+0000","last_undegraded":"2023-12-06T15:31:25.359996+0000","last_fullsized":"2023-12-06T15:31:25.359996+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.360527+0000","last_change":"2023-12-06T15:31:25.360527+0000","last_active":"2023-12-06T15:31:25.360527+0000","last_peered":"2023-12-06T15:31:25.360527+0000","last_clean":"2023-12-06T15:31:25.360527+0000","last_became_active":"2023-12-06T15:31:25.360028+0000","last_became_peered":"2023-12-06T15:31:25.360028+0000","last_unstale":"2023-12-06T15:31:25.360527+0000","last_undegraded":"2023-12-06T15:31:25.360527+0000","last_fullsized":"2023-12-06T15:31:25.360527+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297188,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074652,"statfs":{"total":96636764160,"available":96332443648,"internally_reserved":0,"allocated":94208,"data_stored":24183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074640,"statfs":{"total":96636764160,"available":96332431360,"internally_reserved":0,"allocated":106496,"data_stored":29165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":25253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T15:31:29.268 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-06T15:31:29.553 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:29.555 INFO:teuthology.orchestra.run.smithi176.stderr:dumped all 2023-12-06T15:31:29.566 INFO:teuthology.orchestra.run.smithi176.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-12-06T15:31:28.304502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891580,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223940,"statfs":{"total":289910292480,"available":288997314560,"internally_reserved":0,"allocated":299008,"data_stored":78601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.361264+0000","last_change":"2023-12-06T15:31:25.361264+0000","last_active":"2023-12-06T15:31:25.361264+0000","last_peered":"2023-12-06T15:31:25.361264+0000","last_clean":"2023-12-06T15:31:25.361264+0000","last_became_active":"2023-12-06T15:31:25.360985+0000","last_became_peered":"2023-12-06T15:31:25.360985+0000","last_unstale":"2023-12-06T15:31:25.361264+0000","last_undegraded":"2023-12-06T15:31:25.361264+0000","last_fullsized":"2023-12-06T15:31:25.361264+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.360417+0000","last_change":"2023-12-06T15:31:25.360417+0000","last_active":"2023-12-06T15:31:25.360417+0000","last_peered":"2023-12-06T15:31:25.360417+0000","last_clean":"2023-12-06T15:31:25.360417+0000","last_became_active":"2023-12-06T15:31:25.360097+0000","last_became_peered":"2023-12-06T15:31:25.360097+0000","last_unstale":"2023-12-06T15:31:25.360417+0000","last_undegraded":"2023-12-06T15:31:25.360417+0000","last_fullsized":"2023-12-06T15:31:25.360417+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.360590+0000","last_change":"2023-12-06T15:31:25.360590+0000","last_active":"2023-12-06T15:31:25.360590+0000","last_peered":"2023-12-06T15:31:25.360590+0000","last_clean":"2023-12-06T15:31:25.360590+0000","last_became_active":"2023-12-06T15:31:25.360289+0000","last_became_peered":"2023-12-06T15:31:25.360289+0000","last_unstale":"2023-12-06T15:31:25.360590+0000","last_undegraded":"2023-12-06T15:31:25.360590+0000","last_fullsized":"2023-12-06T15:31:25.360590+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.350181+0000","last_change":"2023-12-06T15:31:25.346677+0000","last_active":"2023-12-06T15:31:25.350181+0000","last_peered":"2023-12-06T15:31:25.350181+0000","last_clean":"2023-12-06T15:31:25.350181+0000","last_became_active":"2023-12-06T15:31:25.346360+0000","last_became_peered":"2023-12-06T15:31:25.346360+0000","last_unstale":"2023-12-06T15:31:25.350181+0000","last_undegraded":"2023-12-06T15:31:25.350181+0000","last_fullsized":"2023-12-06T15:31:25.350181+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.360336+0000","last_change":"2023-12-06T15:31:25.360336+0000","last_active":"2023-12-06T15:31:25.360336+0000","last_peered":"2023-12-06T15:31:25.360336+0000","last_clean":"2023-12-06T15:31:25.360336+0000","last_became_active":"2023-12-06T15:31:25.360016+0000","last_became_peered":"2023-12-06T15:31:25.360016+0000","last_unstale":"2023-12-06T15:31:25.360336+0000","last_undegraded":"2023-12-06T15:31:25.360336+0000","last_fullsized":"2023-12-06T15:31:25.360336+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.345634+0000","last_change":"2023-12-06T15:31:25.345634+0000","last_active":"2023-12-06T15:31:25.345634+0000","last_peered":"2023-12-06T15:31:25.345634+0000","last_clean":"2023-12-06T15:31:25.345634+0000","last_became_active":"2023-12-06T15:31:25.345340+0000","last_became_peered":"2023-12-06T15:31:25.345340+0000","last_unstale":"2023-12-06T15:31:25.345634+0000","last_undegraded":"2023-12-06T15:31:25.345634+0000","last_fullsized":"2023-12-06T15:31:25.345634+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.353041+0000","last_change":"2023-12-06T15:31:25.348740+0000","last_active":"2023-12-06T15:31:25.353041+0000","last_peered":"2023-12-06T15:31:25.353041+0000","last_clean":"2023-12-06T15:31:25.353041+0000","last_became_active":"2023-12-06T15:31:25.348397+0000","last_became_peered":"2023-12-06T15:31:25.348397+0000","last_unstale":"2023-12-06T15:31:25.353041+0000","last_undegraded":"2023-12-06T15:31:25.353041+0000","last_fullsized":"2023-12-06T15:31:25.353041+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.359996+0000","last_change":"2023-12-06T15:31:25.359996+0000","last_active":"2023-12-06T15:31:25.359996+0000","last_peered":"2023-12-06T15:31:25.359996+0000","last_clean":"2023-12-06T15:31:25.359996+0000","last_became_active":"2023-12-06T15:31:25.359616+0000","last_became_peered":"2023-12-06T15:31:25.359616+0000","last_unstale":"2023-12-06T15:31:25.359996+0000","last_undegraded":"2023-12-06T15:31:25.359996+0000","last_fullsized":"2023-12-06T15:31:25.359996+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:25.360527+0000","last_change":"2023-12-06T15:31:25.360527+0000","last_active":"2023-12-06T15:31:25.360527+0000","last_peered":"2023-12-06T15:31:25.360527+0000","last_clean":"2023-12-06T15:31:25.360527+0000","last_became_active":"2023-12-06T15:31:25.360028+0000","last_became_peered":"2023-12-06T15:31:25.360028+0000","last_unstale":"2023-12-06T15:31:25.360527+0000","last_undegraded":"2023-12-06T15:31:25.360527+0000","last_fullsized":"2023-12-06T15:31:25.360527+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:24.335467+0000","last_clean_scrub_stamp":"2023-12-06T15:31:24.335467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297188,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074652,"statfs":{"total":96636764160,"available":96332443648,"internally_reserved":0,"allocated":94208,"data_stored":24183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074640,"statfs":{"total":96636764160,"available":96332431360,"internally_reserved":0,"allocated":106496,"data_stored":29165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":25253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T15:31:29.567 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-06T15:31:29.567 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-12-06T15:31:29.567 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-12-06T15:31:29.567 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-12-06T15:31:29.888 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:29.901 INFO:teuthology.orchestra.run.smithi176.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-06T15:31:29.901 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-12-06T15:31:29.902 INFO:teuthology.run_tasks:Running task ceph... 2023-12-06T15:31:29.911 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-12-06T15:31:29.911 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-06T15:31:29.912 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-06_15:04:09-rbd-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7480288', 'branch': 'wip-yuri6-testing-2023-12-05-0753-pacific', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': None, 'first_in_suite': False, 'job_id': '7480288', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-12-06_15:04:09-rbd-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2023-12-05-0753-pacific'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2023-12-05-0753-pacific', 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5062, 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri6-testing-2023-12-05-0753-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b', 'targets': {'smithi176.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdeHQEar7hPqc96kDyNSh6M7LB7BwlUWoPVhBK4v+HkXWC8SO3dk4cEx/FQYQnJlhdNSE30SESLYGC+N2hMAnc=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD4ib3EayXX/EbckMYpTCjmhZ2CmtXTzz/JTEtV7vSVUI06kQZE5WQuceaANryCFJip34iVZ39qXWIo6HE17Fmo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-06_15:04:09', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-06T15:31:29.912 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-06T15:31:29.915 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-06T15:31:29.938 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2023-12-06T15:31:29.973 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2023-12-06T15:31:30.010 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:30.010 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2023-12-06T15:31:30.070 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-06T15:31:30.071 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2023-12-06T15:31:30.131 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-06T15:31:30.131 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T15:31:30.131 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 397325 Links: 1 2023-12-06T15:31:30.132 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T15:31:30.132 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T15:31:30.132 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2023-12-06 15:30:26.084329067 +0000 2023-12-06T15:31:30.132 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2023-12-06 15:26:19.426873768 +0000 2023-12-06T15:31:30.132 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2023-12-06 15:26:19.426873768 +0000 2023-12-06T15:31:30.132 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2023-12-06T15:31:30.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-06T15:31:30.200 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2023-12-06T15:31:30.200 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2023-12-06T15:31:30.200 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000244411 s, 2.1 MB/s 2023-12-06T15:31:30.202 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-06T15:31:30.263 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2023-12-06T15:31:30.323 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-06T15:31:30.323 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T15:31:30.323 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 382753 Links: 1 2023-12-06T15:31:30.323 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T15:31:30.323 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T15:31:30.323 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2023-12-06 15:30:26.084329067 +0000 2023-12-06T15:31:30.323 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2023-12-06 15:26:19.933866356 +0000 2023-12-06T15:31:30.324 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2023-12-06 15:26:19.933866356 +0000 2023-12-06T15:31:30.324 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2023-12-06T15:31:30.324 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-06T15:31:30.390 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2023-12-06T15:31:30.390 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2023-12-06T15:31:30.390 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000206284 s, 2.5 MB/s 2023-12-06T15:31:30.391 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-06T15:31:30.451 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2023-12-06T15:31:30.510 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-06T15:31:30.510 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T15:31:30.510 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 393411 Links: 1 2023-12-06T15:31:30.510 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T15:31:30.510 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T15:31:30.510 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2023-12-06 15:30:26.084329067 +0000 2023-12-06T15:31:30.510 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2023-12-06 15:26:20.428859119 +0000 2023-12-06T15:31:30.510 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2023-12-06 15:26:20.428859119 +0000 2023-12-06T15:31:30.510 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2023-12-06T15:31:30.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-06T15:31:30.576 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2023-12-06T15:31:30.576 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2023-12-06T15:31:30.576 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000198107 s, 2.6 MB/s 2023-12-06T15:31:30.577 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-06T15:31:30.636 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2023-12-06T15:31:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-06T15:31:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-06T15:31:30.695 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 397185 Links: 1 2023-12-06T15:31:30.695 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-06T15:31:30.695 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2023-12-06T15:31:30.695 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2023-12-06 15:30:26.084329067 +0000 2023-12-06T15:31:30.695 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2023-12-06 15:26:21.008850640 +0000 2023-12-06T15:31:30.695 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2023-12-06 15:26:21.008850640 +0000 2023-12-06T15:31:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2023-12-06T15:31:30.695 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-06T15:31:30.760 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2023-12-06T15:31:30.761 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2023-12-06T15:31:30.761 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000161145 s, 3.2 MB/s 2023-12-06T15:31:30.762 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-06T15:31:30.821 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-06T15:31:30.821 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-06T15:31:30.821 INFO:tasks.ceph:Generating config... 2023-12-06T15:31:30.823 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-06T15:31:30.823 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-06T15:31:30.823 INFO:tasks.ceph:[client] log to stderr = False 2023-12-06T15:31:30.823 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-06T15:31:30.824 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2023-12-06T15:31:30.824 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:30.824 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:30.824 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:30.824 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:30.824 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:30.825 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:30.825 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:30.825 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:30.825 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-06T15:31:30.825 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-06T15:31:30.825 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-06T15:31:30.825 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-06T15:31:30.826 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-06T15:31:30.826 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-06T15:31:30.826 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-06T15:31:30.826 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-06T15:31:30.826 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-06T15:31:30.826 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-06T15:31:30.826 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-06T15:31:30.826 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-06T15:31:30.827 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-06T15:31:30.827 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-06T15:31:30.827 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-06T15:31:30.827 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-06T15:31:30.827 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-06T15:31:30.829 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-06T15:31:30.829 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-06T15:31:30.829 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-06T15:31:30.829 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-06T15:31:30.830 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-06T15:31:30.831 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-12-06T15:31:30.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-12-06T15:31:30.900 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.keyring 2023-12-06T15:31:30.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-12-06T15:31:30.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-06T15:31:31.050 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.204')] 2023-12-06T15:31:31.051 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.204', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-06T15:31:31.051 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:31.051 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-06T15:31:31.108 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.204 --print /home/ubuntu/cephtest/cluster2.monmap 2023-12-06T15:31:31.186 INFO:teuthology.orchestra.run.smithi204.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-12-06T15:31:31.186 INFO:teuthology.orchestra.run.smithi204.stdout:monmaptool: generated fsid b46edd0e-c9dd-4ccc-9a28-d1944f8c6432 2023-12-06T15:31:31.186 INFO:teuthology.orchestra.run.smithi204.stdout:epoch 0 2023-12-06T15:31:31.187 INFO:teuthology.orchestra.run.smithi204.stdout:fsid b46edd0e-c9dd-4ccc-9a28-d1944f8c6432 2023-12-06T15:31:31.187 INFO:teuthology.orchestra.run.smithi204.stdout:last_changed 2023-12-06T15:31:31.186789+0000 2023-12-06T15:31:31.187 INFO:teuthology.orchestra.run.smithi204.stdout:created 2023-12-06T15:31:31.186789+0000 2023-12-06T15:31:31.187 INFO:teuthology.orchestra.run.smithi204.stdout:min_mon_release 0 (unknown) 2023-12-06T15:31:31.187 INFO:teuthology.orchestra.run.smithi204.stdout:election_strategy: 1 2023-12-06T15:31:31.187 INFO:teuthology.orchestra.run.smithi204.stdout:0: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.a 2023-12-06T15:31:31.187 INFO:teuthology.orchestra.run.smithi204.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-12-06T15:31:31.188 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-06T15:31:31.246 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID b46edd0e-c9dd-4ccc-9a28-d1944f8c6432... 2023-12-06T15:31:31.247 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-06T15:31:31.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-06T15:31:31.300 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-06T15:31:31.301 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-06T15:31:31.302 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: mon_allow_pool_size_one = true 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = 172.21.15.204 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: mon client directed command retry = 5 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject socket failures = 5000 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = b46edd0e-c9dd-4ccc-9a28-d1944f8c6432 2023-12-06T15:31:31.303 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2023-12-06T15:31:31.304 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: bdev async discard = True 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: bdev enable discard = True 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore allocator = stupid 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2023-12-06T15:31:31.305 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 1/20 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 1/20 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 4/10 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2023-12-06T15:31:31.306 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2023-12-06T15:31:31.307 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-06T15:31:31.308 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout: debug rbd = 20 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout: debug rbd_mirror = 15 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout: log to stderr = False 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout: rbd default features = 125 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout: rbd mirror image policy type = none 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.0] 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.1] 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.2] 2023-12-06T15:31:31.309 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:31.310 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:31.310 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.3] 2023-12-06T15:31:31.310 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:31.310 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:31.331 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2023-12-06T15:31:31.331 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2023-12-06T15:31:31.331 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-06T15:31:31.331 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2023-12-06T15:31:31.331 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.331 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2023-12-06T15:31:31.331 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.331 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2023-12-06T15:31:31.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2023-12-06T15:31:31.333 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = 172.21.15.204 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon client directed command retry = 5 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject socket failures = 5000 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = b46edd0e-c9dd-4ccc-9a28-d1944f8c6432 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2023-12-06T15:31:31.334 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2023-12-06T15:31:31.335 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: bdev async discard = True 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: bdev enable discard = True 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore allocator = stupid 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore block size = 96636764160 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore fsck on mount = True 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluefs = 1/20 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluestore = 1/20 2023-12-06T15:31:31.336 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: debug rocksdb = 4/10 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd backfillfull_ratio = 0.85 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd full ratio = 0.9 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd nearfull ratio = 0.8 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: osd failsafe full ratio = 0.95 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: osd objectstore = bluestore 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2023-12-06T15:31:31.337 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-06T15:31:31.338 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: debug rbd = 20 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: debug rbd_mirror = 15 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: log to stderr = False 2023-12-06T15:31:31.339 INFO:teuthology.orchestra.run.smithi204.stdout: rbd default features = 125 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout: rbd mirror image policy type = none 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.0] 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.1] 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.2] 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.3] 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-06T15:31:31.340 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-06T15:31:31.341 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-12-06T15:31:31.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-12-06T15:31:31.433 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-06T15:31:31.433 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:31.433 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-12-06T15:31:31.451 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:31.451 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-12-06T15:31:31.509 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2023-12-06T15:31:31.509 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:31.509 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-06T15:31:31.509 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-06T15:31:31.553 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:31.553 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-06T15:31:31.614 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2023-12-06T15:31:31.615 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:31.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-06T15:31:31.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-06T15:31:31.651 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:31.651 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-06T15:31:31.708 INFO:tasks.ceph:Setting up mon nodes... 2023-12-06T15:31:31.709 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-06T15:31:31.709 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-06T15:31:31.766 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-06T15:31:31.768 INFO:tasks.ceph:Setting up mds nodes... 2023-12-06T15:31:31.769 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-06T15:31:31.769 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-12-06T15:31:31.811 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-12-06T15:31:31.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-12-06T15:31:31.862 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-12-06T15:31:31.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-06T15:31:31.953 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-06T15:31:31.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-06T15:31:32.044 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-06T15:31:32.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-06T15:31:32.137 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-06T15:31:32.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-06T15:31:32.229 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-06T15:31:32.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-06T15:31:32.321 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-06T15:31:32.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-06T15:31:32.412 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-06T15:31:32.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-06T15:31:32.503 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-06T15:31:32.515 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-06T15:31:32.515 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-06T15:31:32.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-12-06T15:31:32.582 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-06T15:31:32.582 INFO:tasks.ceph:role: cluster2.osd.0 2023-12-06T15:31:32.582 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com 2023-12-06T15:31:32.582 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-06T15:31:32.656 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T15:31:32.657 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T15:31:32.657 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T15:31:32.657 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-06T15:31:32.657 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T15:31:32.657 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2023-12-06T15:31:32.657 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T15:31:32.657 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-06T15:31:32.657 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T15:31:32.657 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T15:31:33.090 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2023-12-06T15:31:33.092 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2023-12-06T15:31:33.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-12-06T15:31:33.269 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-12-06T15:31:33.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-12-06T15:31:33.371 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-06T15:31:33.371 INFO:tasks.ceph:role: cluster2.osd.1 2023-12-06T15:31:33.371 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com 2023-12-06T15:31:33.371 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-06T15:31:33.445 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T15:31:33.446 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T15:31:33.446 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T15:31:33.446 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-06T15:31:33.446 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T15:31:33.446 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2023-12-06T15:31:33.446 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T15:31:33.446 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-06T15:31:33.446 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T15:31:33.446 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T15:31:33.868 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2023-12-06T15:31:33.870 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2023-12-06T15:31:33.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-12-06T15:31:33.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-12-06T15:31:34.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-12-06T15:31:34.078 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-06T15:31:34.078 INFO:tasks.ceph:role: cluster2.osd.2 2023-12-06T15:31:34.078 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com 2023-12-06T15:31:34.078 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-06T15:31:34.153 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-06T15:31:34.153 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-06T15:31:34.153 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-06T15:31:34.153 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-06T15:31:34.153 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-06T15:31:34.153 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2023-12-06T15:31:34.153 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-06T15:31:34.153 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-06T15:31:34.153 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-06T15:31:34.153 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-06T15:31:34.597 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2023-12-06T15:31:34.598 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2023-12-06T15:31:34.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-12-06T15:31:34.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-12-06T15:31:34.778 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-06T15:31:34.874 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:34.873+0000 7f2762db8380 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-12-06T15:31:34.874 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:34.874+0000 7f2762db8380 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-12-06T15:31:34.874 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:34.874+0000 7f2762db8380 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-06T15:31:34.875 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:34.874+0000 7f2762db8380 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-06T15:31:34.875 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:34.874+0000 7f2762db8380 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-06T15:31:34.881 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:34.881+0000 7f2762db8380 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-12-06T15:31:37.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-12-06T15:31:37.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-06T15:31:37.846 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:37.845+0000 7f169b466380 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-12-06T15:31:37.846 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:37.845+0000 7f169b466380 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-12-06T15:31:37.846 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:37.845+0000 7f169b466380 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-06T15:31:37.846 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:37.845+0000 7f169b466380 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-06T15:31:37.846 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:37.845+0000 7f169b466380 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-06T15:31:37.846 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:37.846+0000 7f169b466380 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-12-06T15:31:40.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-12-06T15:31:40.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-06T15:31:40.809 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:40.809+0000 7f17b3928380 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-12-06T15:31:40.809 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:40.809+0000 7f17b3928380 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-12-06T15:31:40.810 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:40.809+0000 7f17b3928380 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-06T15:31:40.810 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:40.809+0000 7f17b3928380 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-06T15:31:40.810 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:40.809+0000 7f17b3928380 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-06T15:31:40.810 INFO:teuthology.orchestra.run.smithi204.stderr:2023-12-06T15:31:40.809+0000 7f17b3928380 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-12-06T15:31:43.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-12-06T15:31:43.678 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-06T15:31:43.678 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:43.678 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-12-06T15:31:43.713 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:43.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-12-06T15:31:43.747 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:43.747 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-12-06T15:31:43.815 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:43.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-12-06T15:31:43.883 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-06T15:31:43.883 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-12-06T15:31:43.905 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:43.905 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-12-06T15:31:43.941 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:43.941 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-12-06T15:31:43.999 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:44.000 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-12-06T15:31:44.057 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:44.057 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-12-06T15:31:44.114 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:44.115 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-12-06T15:31:44.171 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:44.172 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-12-06T15:31:44.229 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:44.229 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-12-06T15:31:44.287 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:44.287 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-12-06T15:31:44.344 INFO:tasks.ceph:Adding keys to all mons... 2023-12-06T15:31:44.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo tee -a /etc/ceph/cluster2.keyring 2023-12-06T15:31:44.409 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.x] 2023-12-06T15:31:44.409 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDTk3BltGC4LRAAaGfP9vMXgLfSFRfcda26Tg== 2023-12-06T15:31:44.409 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2023-12-06T15:31:44.409 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDWk3Blq7cdNBAAhsR10dbL3sJ/Y2IGjQgWwg== 2023-12-06T15:31:44.409 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2023-12-06T15:31:44.409 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDZk3BlHSZuMhAAsKbIwTmtra7EzvuNuEv/8A== 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDck3Bl72JDMBAATvauRGUkNuoHHIW75XWe5Q== 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDTk3BlNgphMBAAMpv2XSiZjLSN3tO3/gxexw== 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror] 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDTk3BljZpyMxAAdlIxAGIH8e84u/whcbmxGg== 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.0] 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDTk3BlH3nhOBAAa3nL/yQhrdVRtkfno/LUXQ== 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.1] 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDUk3BlsXiwAhAAlne3+pmyCi6ochyallHM7Q== 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.2] 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDUk3Bl+9Y8CBAAVlxQ3vFl7wgRWveQ+vGlIA== 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.3] 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDUk3BlHL+3DRAA5OzBHTvPAlGugJ8rd00QWg== 2023-12-06T15:31:44.410 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.4] 2023-12-06T15:31:44.411 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDUk3BlJFQqExAA7ksEevDJ+gJ/1gc8+ZBBiw== 2023-12-06T15:31:44.411 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.5] 2023-12-06T15:31:44.411 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDUk3BlGAihGBAABabwsjTia/WWwJurkG+J5Q== 2023-12-06T15:31:44.411 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.6] 2023-12-06T15:31:44.411 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDUk3BlVhUOHhAAu1bQaWxc8QeLwR6YfIE3TA== 2023-12-06T15:31:44.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-06T15:31:44.501 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T15:31:44.549 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T15:31:44.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-06T15:31:44.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:44.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:44.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:44.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:44.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:44.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:44.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:45.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:45.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-06T15:31:45.107 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-06T15:31:45.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-12-06T15:31:45.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-12-06T15:31:45.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-12-06T15:31:45.254 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-12-06T15:31:45.312 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-12-06T15:31:45.312 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-12-06T15:31:45.312 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-12-06T15:31:45.355 INFO:tasks.ceph.cluster2.mon.a:Started 2023-12-06T15:31:45.355 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-12-06T15:31:45.355 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-12-06T15:31:45.355 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-12-06T15:31:45.358 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-12-06T15:31:45.358 INFO:tasks.ceph:Setting crush tunables to default 2023-12-06T15:31:45.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd crush tunables default 2023-12-06T15:31:45.563 INFO:teuthology.orchestra.run.smithi204.stderr:adjusted tunables profile to default 2023-12-06T15:31:45.574 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-12-06T15:31:45.574 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:45.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-12-06T15:31:45.595 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:45.592+0000 7fa48d4a4680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-06T15:31:45.601 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:45.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-12-06T15:31:45.654 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:45.653+0000 7fa48d4a4680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-06T15:31:45.668 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:31:45.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-12-06T15:31:45.735 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd new 757cfdd9-6c28-484b-b857-3f8f11ba5d60 0 2023-12-06T15:31:45.921 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:45.921+0000 7fa48d4a4680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-06T15:31:45.974 INFO:teuthology.orchestra.run.smithi204.stdout:0 2023-12-06T15:31:45.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd new 24c76da8-f3f0-4914-98d8-6bbbd2c9892b 1 2023-12-06T15:31:46.184 INFO:teuthology.orchestra.run.smithi204.stdout:1 2023-12-06T15:31:46.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd new 13bd5315-99f5-43d5-b30a-5a6dc358a543 2 2023-12-06T15:31:46.394 INFO:teuthology.orchestra.run.smithi204.stdout:2 2023-12-06T15:31:46.406 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-12-06T15:31:46.406 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-12-06T15:31:46.408 INFO:tasks.ceph.cluster2.osd.0:Started 2023-12-06T15:31:46.409 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-12-06T15:31:46.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-12-06T15:31:46.412 INFO:tasks.ceph.cluster2.osd.1:Started 2023-12-06T15:31:46.413 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-12-06T15:31:46.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-12-06T15:31:46.415 INFO:tasks.ceph.cluster2.osd.2:Started 2023-12-06T15:31:46.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-06T15:31:46.528 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:46.522+0000 7fa48d4a4680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-06T15:31:46.594 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:46.593+0000 7fa48d4a4680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-06T15:31:46.610 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:46.610 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":5,"fsid":"b46edd0e-c9dd-4ccc-9a28-d1944f8c6432","created":"2023-12-06T15:31:45.463841+0000","modified":"2023-12-06T15:31:46.393179+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T15:31:46.393179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"757cfdd9-6c28-484b-b857-3f8f11ba5d60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24c76da8-f3f0-4914-98d8-6bbbd2c9892b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"13bd5315-99f5-43d5-b30a-5a6dc358a543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T15:31:46.622 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-12-06T15:31:46.622 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-06T15:31:46.777 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:46.776+0000 7fa48d4a4680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-06T15:31:46.924 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-06T15:31:47.006 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:31:47.005+0000 7f6eef2eb380 -1 Falling back to public interface 2023-12-06T15:31:47.011 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:31:47.010+0000 7f44c5aa7380 -1 Falling back to public interface 2023-12-06T15:31:47.011 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2023-12-06T15:31:47.011+0000 7f6470454380 -1 Falling back to public interface 2023-12-06T15:31:47.105 INFO:teuthology.misc.health.smithi204.stdout: 2023-12-06T15:31:47.105 INFO:teuthology.misc.health.smithi204.stdout:{"epoch":5,"fsid":"b46edd0e-c9dd-4ccc-9a28-d1944f8c6432","created":"2023-12-06T15:31:45.463841+0000","modified":"2023-12-06T15:31:46.393179+0000","last_up_change":"0.000000","last_in_change":"2023-12-06T15:31:46.393179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"757cfdd9-6c28-484b-b857-3f8f11ba5d60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24c76da8-f3f0-4914-98d8-6bbbd2c9892b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"13bd5315-99f5-43d5-b30a-5a6dc358a543","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T15:31:47.116 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-06T15:31:47.187 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:47.186+0000 7fa48d4a4680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-06T15:31:47.288 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:47.287+0000 7fa48d4a4680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-06T15:31:47.443 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:47.441+0000 7fa48d4a4680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-06T15:31:47.516 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:47.515+0000 7fa48d4a4680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-06T15:31:47.581 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:47.580+0000 7fa48d4a4680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-06T15:31:47.831 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:47.830+0000 7fa48d4a4680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-06T15:31:48.020 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:48.019+0000 7fa48d4a4680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-06T15:31:48.665 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:48.665+0000 7fa48d4a4680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-06T15:31:48.732 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:48.731+0000 7fa48d4a4680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-06T15:31:48.805 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:48.805+0000 7fa48d4a4680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-06T15:31:48.947 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:48.946+0000 7fa48d4a4680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-06T15:31:49.014 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:49.013+0000 7fa48d4a4680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-06T15:31:49.127 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:49.126+0000 7fa48d4a4680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-06T15:31:49.232 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:49.231+0000 7fa48d4a4680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-06T15:31:49.525 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:49.525+0000 7fa48d4a4680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-06T15:31:49.593 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:49.592+0000 7fa48d4a4680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-06T15:31:50.422 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:31:50.422+0000 7f6eef2eb380 -1 osd.0 0 log_to_monitors {default=true} 2023-12-06T15:31:50.485 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2023-12-06T15:31:50.485+0000 7f6470454380 -1 osd.1 0 log_to_monitors {default=true} 2023-12-06T15:31:50.487 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:31:50.487+0000 7f44c5aa7380 -1 osd.2 0 log_to_monitors {default=true} 2023-12-06T15:31:50.613 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T15:31:50.612+0000 7fa4429fd700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-06T15:31:51.669 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:31:51.669+0000 7f6ee748f700 -1 osd.0 0 waiting for initial osdmap 2023-12-06T15:31:51.670 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2023-12-06T15:31:51.670+0000 7f6466df5700 -1 osd.1 0 waiting for initial osdmap 2023-12-06T15:31:51.671 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:31:51.671+0000 7f44bc448700 -1 osd.2 0 waiting for initial osdmap 2023-12-06T15:31:51.673 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:31:51.673+0000 7f6ee2408700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T15:31:51.674 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2023-12-06T15:31:51.674+0000 7f6463571700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T15:31:51.675 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:31:51.675+0000 7f44b8bc4700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-06T15:31:53.420 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-06T15:31:53.713 INFO:teuthology.misc.health.smithi204.stdout: 2023-12-06T15:31:53.714 INFO:teuthology.misc.health.smithi204.stdout:{"epoch":9,"fsid":"b46edd0e-c9dd-4ccc-9a28-d1944f8c6432","created":"2023-12-06T15:31:45.463841+0000","modified":"2023-12-06T15:31:53.669527+0000","last_up_change":"2023-12-06T15:31:52.665787+0000","last_in_change":"2023-12-06T15:31:46.393179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-06T15:31:52.704987+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"757cfdd9-6c28-484b-b857-3f8f11ba5d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":115494},{"type":"v1","addr":"172.21.15.204:6801","nonce":115494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":115494},{"type":"v1","addr":"172.21.15.204:6803","nonce":115494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":115494},{"type":"v1","addr":"172.21.15.204:6807","nonce":115494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":115494},{"type":"v1","addr":"172.21.15.204:6805","nonce":115494}]},"public_addr":"172.21.15.204:6801/115494","cluster_addr":"172.21.15.204:6803/115494","heartbeat_back_addr":"172.21.15.204:6807/115494","heartbeat_front_addr":"172.21.15.204:6805/115494","state":["exists","up"]},{"osd":1,"uuid":"24c76da8-f3f0-4914-98d8-6bbbd2c9892b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":115495},{"type":"v1","addr":"172.21.15.204:6814","nonce":115495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":115495},{"type":"v1","addr":"172.21.15.204:6818","nonce":115495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":115495},{"type":"v1","addr":"172.21.15.204:6823","nonce":115495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":115495},{"type":"v1","addr":"172.21.15.204:6821","nonce":115495}]},"public_addr":"172.21.15.204:6814/115495","cluster_addr":"172.21.15.204:6818/115495","heartbeat_back_addr":"172.21.15.204:6823/115495","heartbeat_front_addr":"172.21.15.204:6821/115495","state":["exists","up"]},{"osd":2,"uuid":"13bd5315-99f5-43d5-b30a-5a6dc358a543","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":115496},{"type":"v1","addr":"172.21.15.204:6809","nonce":115496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":115496},{"type":"v1","addr":"172.21.15.204:6811","nonce":115496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6817","nonce":115496},{"type":"v1","addr":"172.21.15.204:6819","nonce":115496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":115496},{"type":"v1","addr":"172.21.15.204:6815","nonce":115496}]},"public_addr":"172.21.15.204:6809/115496","cluster_addr":"172.21.15.204:6811/115496","heartbeat_back_addr":"172.21.15.204:6819/115496","heartbeat_front_addr":"172.21.15.204:6815/115496","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:51.439505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:51.495811+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:51.531309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T15:31:53.726 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-06T15:31:53.726 INFO:tasks.ceph:Creating RBD pool 2023-12-06T15:31:53.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-12-06T15:31:54.683 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'rbd' created 2023-12-06T15:31:54.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-06T15:31:55.682 INFO:teuthology.orchestra.run.smithi204.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-06T15:31:55.697 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-12-06T15:31:55.698 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-12-06T15:31:55.698 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-12-06T15:31:55.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-12-06T15:31:55.698 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-06T15:31:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:56.044 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":4,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":118396},{"type":"v1","addr":"172.21.15.176:6827","nonce":118396}]},"active_addr":"172.21.15.176:6827/118396","active_change":"2023-12-06T15:31:49.607259+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":1306654844}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":3293610099}]}]} 2023-12-06T15:31:56.046 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-12-06T15:31:56.046 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-12-06T15:31:56.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-06T15:31:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:56.333 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":11,"fsid":"b46edd0e-c9dd-4ccc-9a28-d1944f8c6432","created":"2023-12-06T15:31:45.463841+0000","modified":"2023-12-06T15:31:55.673447+0000","last_up_change":"2023-12-06T15:31:52.665787+0000","last_in_change":"2023-12-06T15:31:46.393179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-06T15:31:52.704987+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T15:31:54.043280+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"757cfdd9-6c28-484b-b857-3f8f11ba5d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":115494},{"type":"v1","addr":"172.21.15.204:6801","nonce":115494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":115494},{"type":"v1","addr":"172.21.15.204:6803","nonce":115494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":115494},{"type":"v1","addr":"172.21.15.204:6807","nonce":115494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":115494},{"type":"v1","addr":"172.21.15.204:6805","nonce":115494}]},"public_addr":"172.21.15.204:6801/115494","cluster_addr":"172.21.15.204:6803/115494","heartbeat_back_addr":"172.21.15.204:6807/115494","heartbeat_front_addr":"172.21.15.204:6805/115494","state":["exists","up"]},{"osd":1,"uuid":"24c76da8-f3f0-4914-98d8-6bbbd2c9892b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":115495},{"type":"v1","addr":"172.21.15.204:6814","nonce":115495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":115495},{"type":"v1","addr":"172.21.15.204:6818","nonce":115495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":115495},{"type":"v1","addr":"172.21.15.204:6823","nonce":115495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":115495},{"type":"v1","addr":"172.21.15.204:6821","nonce":115495}]},"public_addr":"172.21.15.204:6814/115495","cluster_addr":"172.21.15.204:6818/115495","heartbeat_back_addr":"172.21.15.204:6823/115495","heartbeat_front_addr":"172.21.15.204:6821/115495","state":["exists","up"]},{"osd":2,"uuid":"13bd5315-99f5-43d5-b30a-5a6dc358a543","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":115496},{"type":"v1","addr":"172.21.15.204:6809","nonce":115496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":115496},{"type":"v1","addr":"172.21.15.204:6811","nonce":115496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6817","nonce":115496},{"type":"v1","addr":"172.21.15.204:6819","nonce":115496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":115496},{"type":"v1","addr":"172.21.15.204:6815","nonce":115496}]},"public_addr":"172.21.15.204:6809/115496","cluster_addr":"172.21.15.204:6811/115496","heartbeat_back_addr":"172.21.15.204:6819/115496","heartbeat_front_addr":"172.21.15.204:6815/115496","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:51.439505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:51.495811+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:51.531309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T15:31:56.346 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-12-06T15:31:56.346 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-06T15:31:56.633 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:56.633 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":11,"fsid":"b46edd0e-c9dd-4ccc-9a28-d1944f8c6432","created":"2023-12-06T15:31:45.463841+0000","modified":"2023-12-06T15:31:55.673447+0000","last_up_change":"2023-12-06T15:31:52.665787+0000","last_in_change":"2023-12-06T15:31:46.393179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-06T15:31:52.704987+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-06T15:31:54.043280+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"757cfdd9-6c28-484b-b857-3f8f11ba5d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":115494},{"type":"v1","addr":"172.21.15.204:6801","nonce":115494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":115494},{"type":"v1","addr":"172.21.15.204:6803","nonce":115494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":115494},{"type":"v1","addr":"172.21.15.204:6807","nonce":115494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":115494},{"type":"v1","addr":"172.21.15.204:6805","nonce":115494}]},"public_addr":"172.21.15.204:6801/115494","cluster_addr":"172.21.15.204:6803/115494","heartbeat_back_addr":"172.21.15.204:6807/115494","heartbeat_front_addr":"172.21.15.204:6805/115494","state":["exists","up"]},{"osd":1,"uuid":"24c76da8-f3f0-4914-98d8-6bbbd2c9892b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":115495},{"type":"v1","addr":"172.21.15.204:6814","nonce":115495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":115495},{"type":"v1","addr":"172.21.15.204:6818","nonce":115495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":115495},{"type":"v1","addr":"172.21.15.204:6823","nonce":115495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":115495},{"type":"v1","addr":"172.21.15.204:6821","nonce":115495}]},"public_addr":"172.21.15.204:6814/115495","cluster_addr":"172.21.15.204:6818/115495","heartbeat_back_addr":"172.21.15.204:6823/115495","heartbeat_front_addr":"172.21.15.204:6821/115495","state":["exists","up"]},{"osd":2,"uuid":"13bd5315-99f5-43d5-b30a-5a6dc358a543","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":115496},{"type":"v1","addr":"172.21.15.204:6809","nonce":115496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":115496},{"type":"v1","addr":"172.21.15.204:6811","nonce":115496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6817","nonce":115496},{"type":"v1","addr":"172.21.15.204:6819","nonce":115496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":115496},{"type":"v1","addr":"172.21.15.204:6815","nonce":115496}]},"public_addr":"172.21.15.204:6809/115496","cluster_addr":"172.21.15.204:6811/115496","heartbeat_back_addr":"172.21.15.204:6819/115496","heartbeat_front_addr":"172.21.15.204:6815/115496","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:51.439505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:51.495811+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-06T15:31:51.531309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-06T15:31:56.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-12-06T15:31:56.794 INFO:teuthology.orchestra.run.smithi204.stdout:34359738372 2023-12-06T15:31:56.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-12-06T15:31:56.941 INFO:teuthology.orchestra.run.smithi204.stdout:34359738372 2023-12-06T15:31:56.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-12-06T15:31:57.089 INFO:teuthology.orchestra.run.smithi204.stdout:34359738372 2023-12-06T15:31:57.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-06T15:31:57.378 INFO:teuthology.orchestra.run.smithi204.stdout:34359738371 2023-12-06T15:31:57.391 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.0 2023-12-06T15:31:58.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-06T15:31:58.695 INFO:teuthology.orchestra.run.smithi204.stdout:34359738372 2023-12-06T15:31:58.709 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.0 2023-12-06T15:31:58.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-06T15:31:58.994 INFO:teuthology.orchestra.run.smithi204.stdout:34359738372 2023-12-06T15:31:59.008 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.1 2023-12-06T15:31:59.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-06T15:31:59.295 INFO:teuthology.orchestra.run.smithi204.stdout:34359738372 2023-12-06T15:31:59.309 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.2 2023-12-06T15:31:59.309 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-06T15:31:59.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-06T15:31:59.593 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:59.595 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2023-12-06T15:31:59.605 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-12-06T15:31:57.612018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891572,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223948,"statfs":{"total":289910292480,"available":288997322752,"internally_reserved":0,"allocated":290816,"data_stored":77406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.686360+0000","last_change":"2023-12-06T15:31:55.686360+0000","last_active":"2023-12-06T15:31:55.686360+0000","last_peered":"2023-12-06T15:31:55.686360+0000","last_clean":"2023-12-06T15:31:55.686360+0000","last_became_active":"2023-12-06T15:31:55.686042+0000","last_became_peered":"2023-12-06T15:31:55.686042+0000","last_unstale":"2023-12-06T15:31:55.686360+0000","last_undegraded":"2023-12-06T15:31:55.686360+0000","last_fullsized":"2023-12-06T15:31:55.686360+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.686604+0000","last_change":"2023-12-06T15:31:55.686604+0000","last_active":"2023-12-06T15:31:55.686604+0000","last_peered":"2023-12-06T15:31:55.686604+0000","last_clean":"2023-12-06T15:31:55.686604+0000","last_became_active":"2023-12-06T15:31:55.685818+0000","last_became_peered":"2023-12-06T15:31:55.685818+0000","last_unstale":"2023-12-06T15:31:55.686604+0000","last_undegraded":"2023-12-06T15:31:55.686604+0000","last_fullsized":"2023-12-06T15:31:55.686604+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.686041+0000","last_change":"2023-12-06T15:31:55.686041+0000","last_active":"2023-12-06T15:31:55.686041+0000","last_peered":"2023-12-06T15:31:55.686041+0000","last_clean":"2023-12-06T15:31:55.686041+0000","last_became_active":"2023-12-06T15:31:55.685746+0000","last_became_peered":"2023-12-06T15:31:55.685746+0000","last_unstale":"2023-12-06T15:31:55.686041+0000","last_undegraded":"2023-12-06T15:31:55.686041+0000","last_fullsized":"2023-12-06T15:31:55.686041+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.684786+0000","last_change":"2023-12-06T15:31:55.684786+0000","last_active":"2023-12-06T15:31:55.684786+0000","last_peered":"2023-12-06T15:31:55.684786+0000","last_clean":"2023-12-06T15:31:55.684786+0000","last_became_active":"2023-12-06T15:31:55.684527+0000","last_became_peered":"2023-12-06T15:31:55.684527+0000","last_unstale":"2023-12-06T15:31:55.684786+0000","last_undegraded":"2023-12-06T15:31:55.684786+0000","last_fullsized":"2023-12-06T15:31:55.684786+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-06T15:31:54.686143+0000","last_change":"2023-12-06T15:31:54.684449+0000","last_active":"2023-12-06T15:31:54.686143+0000","last_peered":"2023-12-06T15:31:54.686143+0000","last_clean":"2023-12-06T15:31:54.686143+0000","last_became_active":"2023-12-06T15:31:54.684109+0000","last_became_peered":"2023-12-06T15:31:54.684109+0000","last_unstale":"2023-12-06T15:31:54.686143+0000","last_undegraded":"2023-12-06T15:31:54.686143+0000","last_fullsized":"2023-12-06T15:31:54.686143+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:53.669527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:53.669527+0000","last_clean_scrub_stamp":"2023-12-06T15:31:53.669527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.685799+0000","last_change":"2023-12-06T15:31:55.685799+0000","last_active":"2023-12-06T15:31:55.685799+0000","last_peered":"2023-12-06T15:31:55.685799+0000","last_clean":"2023-12-06T15:31:55.685799+0000","last_became_active":"2023-12-06T15:31:55.685498+0000","last_became_peered":"2023-12-06T15:31:55.685498+0000","last_unstale":"2023-12-06T15:31:55.685799+0000","last_undegraded":"2023-12-06T15:31:55.685799+0000","last_fullsized":"2023-12-06T15:31:55.685799+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.684647+0000","last_change":"2023-12-06T15:31:55.684647+0000","last_active":"2023-12-06T15:31:55.684647+0000","last_peered":"2023-12-06T15:31:55.684647+0000","last_clean":"2023-12-06T15:31:55.684647+0000","last_became_active":"2023-12-06T15:31:55.684376+0000","last_became_peered":"2023-12-06T15:31:55.684376+0000","last_unstale":"2023-12-06T15:31:55.684647+0000","last_undegraded":"2023-12-06T15:31:55.684647+0000","last_fullsized":"2023-12-06T15:31:55.684647+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.687842+0000","last_change":"2023-12-06T15:31:55.683936+0000","last_active":"2023-12-06T15:31:55.687842+0000","last_peered":"2023-12-06T15:31:55.687842+0000","last_clean":"2023-12-06T15:31:55.687842+0000","last_became_active":"2023-12-06T15:31:55.683661+0000","last_became_peered":"2023-12-06T15:31:55.683661+0000","last_unstale":"2023-12-06T15:31:55.687842+0000","last_undegraded":"2023-12-06T15:31:55.687842+0000","last_fullsized":"2023-12-06T15:31:55.687842+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.684412+0000","last_change":"2023-12-06T15:31:55.684412+0000","last_active":"2023-12-06T15:31:55.684412+0000","last_peered":"2023-12-06T15:31:55.684412+0000","last_clean":"2023-12-06T15:31:55.684412+0000","last_became_active":"2023-12-06T15:31:55.683959+0000","last_became_peered":"2023-12-06T15:31:55.683959+0000","last_unstale":"2023-12-06T15:31:55.684412+0000","last_undegraded":"2023-12-06T15:31:55.684412+0000","last_fullsized":"2023-12-06T15:31:55.684412+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297188,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074652,"statfs":{"total":96636764160,"available":96332443648,"internally_reserved":0,"allocated":94208,"data_stored":25156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T15:31:59.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-06T15:31:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:31:59.891 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2023-12-06T15:31:59.902 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-12-06T15:31:59.612324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891580,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223940,"statfs":{"total":289910292480,"available":288997314560,"internally_reserved":0,"allocated":299008,"data_stored":80935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.686360+0000","last_change":"2023-12-06T15:31:55.686360+0000","last_active":"2023-12-06T15:31:55.686360+0000","last_peered":"2023-12-06T15:31:55.686360+0000","last_clean":"2023-12-06T15:31:55.686360+0000","last_became_active":"2023-12-06T15:31:55.686042+0000","last_became_peered":"2023-12-06T15:31:55.686042+0000","last_unstale":"2023-12-06T15:31:55.686360+0000","last_undegraded":"2023-12-06T15:31:55.686360+0000","last_fullsized":"2023-12-06T15:31:55.686360+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.686604+0000","last_change":"2023-12-06T15:31:55.686604+0000","last_active":"2023-12-06T15:31:55.686604+0000","last_peered":"2023-12-06T15:31:55.686604+0000","last_clean":"2023-12-06T15:31:55.686604+0000","last_became_active":"2023-12-06T15:31:55.685818+0000","last_became_peered":"2023-12-06T15:31:55.685818+0000","last_unstale":"2023-12-06T15:31:55.686604+0000","last_undegraded":"2023-12-06T15:31:55.686604+0000","last_fullsized":"2023-12-06T15:31:55.686604+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.686041+0000","last_change":"2023-12-06T15:31:55.686041+0000","last_active":"2023-12-06T15:31:55.686041+0000","last_peered":"2023-12-06T15:31:55.686041+0000","last_clean":"2023-12-06T15:31:55.686041+0000","last_became_active":"2023-12-06T15:31:55.685746+0000","last_became_peered":"2023-12-06T15:31:55.685746+0000","last_unstale":"2023-12-06T15:31:55.686041+0000","last_undegraded":"2023-12-06T15:31:55.686041+0000","last_fullsized":"2023-12-06T15:31:55.686041+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.684786+0000","last_change":"2023-12-06T15:31:55.684786+0000","last_active":"2023-12-06T15:31:55.684786+0000","last_peered":"2023-12-06T15:31:55.684786+0000","last_clean":"2023-12-06T15:31:55.684786+0000","last_became_active":"2023-12-06T15:31:55.684527+0000","last_became_peered":"2023-12-06T15:31:55.684527+0000","last_unstale":"2023-12-06T15:31:55.684786+0000","last_undegraded":"2023-12-06T15:31:55.684786+0000","last_fullsized":"2023-12-06T15:31:55.684786+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-06T15:31:54.686143+0000","last_change":"2023-12-06T15:31:54.684449+0000","last_active":"2023-12-06T15:31:54.686143+0000","last_peered":"2023-12-06T15:31:54.686143+0000","last_clean":"2023-12-06T15:31:54.686143+0000","last_became_active":"2023-12-06T15:31:54.684109+0000","last_became_peered":"2023-12-06T15:31:54.684109+0000","last_unstale":"2023-12-06T15:31:54.686143+0000","last_undegraded":"2023-12-06T15:31:54.686143+0000","last_fullsized":"2023-12-06T15:31:54.686143+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:53.669527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:53.669527+0000","last_clean_scrub_stamp":"2023-12-06T15:31:53.669527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.685799+0000","last_change":"2023-12-06T15:31:55.685799+0000","last_active":"2023-12-06T15:31:55.685799+0000","last_peered":"2023-12-06T15:31:55.685799+0000","last_clean":"2023-12-06T15:31:55.685799+0000","last_became_active":"2023-12-06T15:31:55.685498+0000","last_became_peered":"2023-12-06T15:31:55.685498+0000","last_unstale":"2023-12-06T15:31:55.685799+0000","last_undegraded":"2023-12-06T15:31:55.685799+0000","last_fullsized":"2023-12-06T15:31:55.685799+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.684647+0000","last_change":"2023-12-06T15:31:55.684647+0000","last_active":"2023-12-06T15:31:55.684647+0000","last_peered":"2023-12-06T15:31:55.684647+0000","last_clean":"2023-12-06T15:31:55.684647+0000","last_became_active":"2023-12-06T15:31:55.684376+0000","last_became_peered":"2023-12-06T15:31:55.684376+0000","last_unstale":"2023-12-06T15:31:55.684647+0000","last_undegraded":"2023-12-06T15:31:55.684647+0000","last_fullsized":"2023-12-06T15:31:55.684647+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.687842+0000","last_change":"2023-12-06T15:31:55.683936+0000","last_active":"2023-12-06T15:31:55.687842+0000","last_peered":"2023-12-06T15:31:55.687842+0000","last_clean":"2023-12-06T15:31:55.687842+0000","last_became_active":"2023-12-06T15:31:55.683661+0000","last_became_peered":"2023-12-06T15:31:55.683661+0000","last_unstale":"2023-12-06T15:31:55.687842+0000","last_undegraded":"2023-12-06T15:31:55.687842+0000","last_fullsized":"2023-12-06T15:31:55.687842+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-06T15:31:55.684412+0000","last_change":"2023-12-06T15:31:55.684412+0000","last_active":"2023-12-06T15:31:55.684412+0000","last_peered":"2023-12-06T15:31:55.684412+0000","last_clean":"2023-12-06T15:31:55.684412+0000","last_became_active":"2023-12-06T15:31:55.683959+0000","last_became_peered":"2023-12-06T15:31:55.683959+0000","last_unstale":"2023-12-06T15:31:55.684412+0000","last_undegraded":"2023-12-06T15:31:55.684412+0000","last_fullsized":"2023-12-06T15:31:55.684412+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-06T15:31:54.672580+0000","last_clean_scrub_stamp":"2023-12-06T15:31:54.672580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297196,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074644,"statfs":{"total":96636764160,"available":96332435456,"internally_reserved":0,"allocated":102400,"data_stored":28685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-06T15:31:59.903 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-12-06T15:31:59.903 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-12-06T15:31:59.903 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-12-06T15:31:59.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-12-06T15:32:00.224 INFO:teuthology.orchestra.run.smithi204.stdout: 2023-12-06T15:32:00.238 INFO:teuthology.orchestra.run.smithi204.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-06T15:32:00.238 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-12-06T15:32:00.239 INFO:teuthology.run_tasks:Running task exec... 2023-12-06T15:32:00.263 INFO:teuthology.task.exec:Executing custom commands... 2023-12-06T15:32:00.263 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi204.front.sepia.ceph.com 2023-12-06T15:32:00.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-06T15:32:00.627 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror 2023-12-06T15:32:00.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-06T15:32:01.001 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.0 2023-12-06T15:32:01.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-06T15:32:01.377 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.1 2023-12-06T15:32:01.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-06T15:32:01.753 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.2 2023-12-06T15:32:01.764 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-06T15:32:02.124 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.3 2023-12-06T15:32:02.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-06T15:32:02.497 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror 2023-12-06T15:32:02.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-06T15:32:02.872 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.0 2023-12-06T15:32:02.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-06T15:32:03.243 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.1 2023-12-06T15:32:03.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-06T15:32:03.618 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.2 2023-12-06T15:32:03.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-06T15:32:03.997 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.3 2023-12-06T15:32:04.009 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-06T15:32:04.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-06T15:32:04.022 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-06T15:32:04.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-06T15:32:04.024 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-06T15:32:04.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-06T15:32:04.032 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-06T15:32:04.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-06T15:32:04.034 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-06T15:32:04.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-06T15:32:04.042 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-06T15:32:04.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-06T15:32:04.045 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-06T15:32:04.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-06T15:32:04.053 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-06T15:32:04.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-06T15:32:04.056 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-06T15:32:04.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-06T15:32:04.065 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-06T15:32:04.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-06T15:32:04.067 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-06T15:32:04.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-06T15:32:04.077 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-06T15:32:04.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-06T15:32:04.079 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-06T15:32:04.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-06T15:32:04.088 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-06T15:32:04.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-06T15:32:04.091 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-06T15:32:04.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-06T15:32:04.101 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-06T15:32:04.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-06T15:32:04.106 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-06T15:32:04.154 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1701876724 2023-12-06T15:32:04.154 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-06T15:32:04.154 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-06T15:32:04.165 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1701876724 2023-12-06T15:32:04.165 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-06T15:32:04.165 INFO:teuthology.run_tasks:Running task workunit... 2023-12-06T15:32:04.178 INFO:tasks.workunit:Pulling workunits from ref d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T15:32:04.178 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-06T15:32:04.179 DEBUG:teuthology.orchestra.run.smithi204:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-06T15:32:04.179 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-12-06T15:32:04.179 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2023-12-06T15:32:04.179 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-12-06T15:32:04.180 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2023-12-06T15:32:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:32:04.199 INFO:teuthology.orchestra.run.smithi204.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-12-06T15:32:04.200 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-06T15:32:04.258 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-06T15:32:04.258 DEBUG:teuthology.orchestra.run.smithi204:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-12-06T15:32:04.316 INFO:tasks.workunit:timeout=6h 2023-12-06T15:32:04.316 INFO:tasks.workunit:cleanup=True 2023-12-06T15:32:04.316 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T15:32:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-12-06T15:33:04.180 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5134144778822759) > weight (0.25) 2023-12-06T15:33:04.181 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8698937621054684) > weight (0.25) 2023-12-06T15:33:04.181 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4093238826129586) > weight (0.25) 2023-12-06T15:33:04.181 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5142982531639396) > weight (0.25) 2023-12-06T15:33:04.181 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2023-12-06T15:33:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Note: switching to 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'. 2023-12-06T15:33:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:state without impacting any branches by switching back to a branch. 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: git switch -c 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Or undo this operation with: 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: git switch - 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-06T15:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:33:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:HEAD is now at d03fca4f35d Merge branch 'wip-ifed-fix-63606' of https://github.com/ifed01/ceph into wip-yuri6-testing-2023-12-05-0753-pacific 2023-12-06T15:33:19.030 DEBUG:teuthology.orchestra.run.smithi204:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-12-06T15:33:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-06T15:33:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-06T15:33:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-06T15:33:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-06T15:33:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-06T15:33:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-06T15:33:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-06T15:33:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-06T15:33:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-06T15:33:19.795 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2023-12-06T15:33:19.795 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-12-06T15:33:19.853 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-12-06T15:33:19.855 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-12-06T15:33:19.855 DEBUG:teuthology.orchestra.run.smithi204:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d03fca4f35d43f61b4679936aef7bd2f01cced0b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-06T15:33:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ IMAGE_COUNT=50 2023-12-06T15:33:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ export LOCKDEP=0 2023-12-06T15:33:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ LOCKDEP=0 2023-12-06T15:33:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-06T15:33:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-12-06T15:33:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ type xmlstarlet 2023-12-06T15:33:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ XMLSTARLET=xmlstarlet 2023-12-06T15:33:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ RBD_MIRROR_INSTANCES=4 2023-12-06T15:33:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CLUSTER1=cluster1 2023-12-06T15:33:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CLUSTER2=cluster2 2023-12-06T15:33:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ PEER_CLUSTER_SUFFIX= 2023-12-06T15:33:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ POOL=mirror 2023-12-06T15:33:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ PARENT_POOL=mirror_parent 2023-12-06T15:33:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ NS1=ns1 2023-12-06T15:33:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ NS2=ns2 2023-12-06T15:33:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ TEMPDIR= 2023-12-06T15:33:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ID=mirror 2023-12-06T15:33:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2023-12-06T15:33:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2023-12-06T15:33:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ MIRROR_POOL_MODE=image 2023-12-06T15:33:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ MIRROR_IMAGE_MODE=snapshot 2023-12-06T15:33:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ export 'CEPH_ARGS=--id mirror' 2023-12-06T15:33:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS='--id mirror' 2023-12-06T15:33:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ LAST_MIRROR_INSTANCE=3 2023-12-06T15:33:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-06T15:33:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-12-06T15:33:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_BIN=. 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_SRC=. 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' -e CMakeCache.txt ']' 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 -gt 0 ']' 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ setup 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local c 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ trap 'cleanup $?' INT TERM EXIT 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ setup_tempdir 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mktemp -d 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ TEMPDIR=/tmp/tmp.Q1BJHHgE3s 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z 1 ']' 2023-12-06T15:33:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ setup_pools cluster1 cluster2 2023-12-06T15:33:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:33:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster2 2023-12-06T15:33:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mon_map_file 2023-12-06T15:33:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mon_addr 2023-12-06T15:33:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local admin_key_file 2023-12-06T15:33:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid 2023-12-06T15:33:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:33:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2023-12-06T15:33:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror' already exists 2023-12-06T15:33:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:33:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-12-06T15:33:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror_parent' already exists 2023-12-06T15:33:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:33:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 pool init mirror 2023-12-06T15:33:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:33:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2023-12-06T15:33:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:33:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:33:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-12-06T15:33:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-12-06T15:33:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-12-06T15:33:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-12-06T15:33:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-12-06T15:33:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-12-06T15:33:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z ']' 2023-12-06T15:33:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z ']' 2023-12-06T15:33:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster1 mirror cluster2 2023-12-06T15:33:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:33:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:33:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster2 2023-12-06T15:33:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster2 2023-12-06T15:33:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2023-12-06T15:33:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:33:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2023-12-06T15:33:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2023-12-06T15:33:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2023-12-06T15:33:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:33:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-12-06T15:33:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=171ac006-ef66-4677-9f7f-b79571e89b7d 2023-12-06T15:33:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2023-12-06T15:33:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:33:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2023-12-06T15:33:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2023-12-06T15:33:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:33:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:33:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-12-06T15:33:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:33:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2023-12-06T15:33:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster2 2023-12-06T15:33:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster2 2023-12-06T15:33:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2023-12-06T15:33:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:33:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2023-12-06T15:33:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2023-12-06T15:33:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2023-12-06T15:33:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:33:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-12-06T15:33:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=d6f3ff8c-5996-4612-860a-32288d13eb46 2023-12-06T15:33:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2023-12-06T15:33:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:33:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2023-12-06T15:33:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2023-12-06T15:33:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:33:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:33:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ setup_pools cluster2 cluster1 2023-12-06T15:33:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:33:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster1 2023-12-06T15:33:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mon_map_file 2023-12-06T15:33:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mon_addr 2023-12-06T15:33:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local admin_key_file 2023-12-06T15:33:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid 2023-12-06T15:33:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:33:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2023-12-06T15:33:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror' already exists 2023-12-06T15:33:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:33:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-12-06T15:33:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror_parent' already exists 2023-12-06T15:33:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:33:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 pool init mirror 2023-12-06T15:33:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:33:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2023-12-06T15:33:34.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T15:33:34.210+0000 7f6db4cb96c0 -1 rbd::mirror::PoolReplayer: 0x55ccc8a44000 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-06T15:33:34.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T15:33:34.216+0000 7fbf510986c0 -1 rbd::mirror::PoolReplayer: 0x5575be12a000 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-06T15:33:34.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi204.stderr:2023-12-06T15:33:34.218+0000 7febf57756c0 -1 rbd::mirror::PoolReplayer: 0x55c3824a2000 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-06T15:33:34.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:33:34.223+0000 7f7186dc46c0 -1 rbd::mirror::PoolReplayer: 0x559414618000 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-06T15:33:34.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T15:33:34.451+0000 7f6db4cb96c0 -1 rbd::mirror::PoolReplayer: 0x55ccca146c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-06T15:33:34.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi204.stderr:2023-12-06T15:33:34.461+0000 7febf57756c0 -1 rbd::mirror::PoolReplayer: 0x55c383b86c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-06T15:33:34.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T15:33:34.461+0000 7fbf510986c0 -1 rbd::mirror::PoolReplayer: 0x5575bf826c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-06T15:33:34.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:33:34.468+0000 7f7186dc46c0 -1 rbd::mirror::PoolReplayer: 0x559415d06c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-06T15:33:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:33:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:33:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-12-06T15:33:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-12-06T15:33:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-12-06T15:33:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-12-06T15:33:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-12-06T15:33:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-12-06T15:33:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z ']' 2023-12-06T15:33:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z ']' 2023-12-06T15:33:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster2 mirror cluster1 2023-12-06T15:33:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:33:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:33:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster1 2023-12-06T15:33:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster1 2023-12-06T15:33:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2023-12-06T15:33:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:33:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2023-12-06T15:33:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2023-12-06T15:33:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2023-12-06T15:33:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:33:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-12-06T15:33:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=f0b6748f-178c-4398-bb28-c00512cef43c 2023-12-06T15:33:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2023-12-06T15:33:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:33:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2023-12-06T15:33:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2023-12-06T15:33:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:33:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:33:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-12-06T15:33:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:33:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2023-12-06T15:33:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster1 2023-12-06T15:33:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster1 2023-12-06T15:33:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2023-12-06T15:33:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:33:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2023-12-06T15:33:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2023-12-06T15:33:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2023-12-06T15:33:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:33:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-12-06T15:33:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=dba51aa0-f2cd-4a94-8c1a-1097bb7aaf81 2023-12-06T15:33:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2023-12-06T15:33:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:33:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2023-12-06T15:33:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2023-12-06T15:33:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:33:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:33:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:33:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster1 2023-12-06T15:33:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:33:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 3 2023-12-06T15:33:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:0 2023-12-06T15:33:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2023-12-06T15:33:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2023-12-06T15:33:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:33:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2023-12-06T15:33:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2023-12-06T15:33:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2023-12-06T15:33:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2023-12-06T15:33:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:1 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2023-12-06T15:33:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:2 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:2 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:2 2023-12-06T15:33:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=2 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 2 = cluster1:2 ']' 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=2 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=2 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:3 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:3 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2023-12-06T15:33:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:3 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=3 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 3 = cluster1:3 ']' 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=3 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=3 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster2 2023-12-06T15:33:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:33:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 3 2023-12-06T15:33:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster2:0 2023-12-06T15:33:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2023-12-06T15:33:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2023-12-06T15:33:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-12-06T15:33:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2023-12-06T15:33:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2023-12-06T15:33:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2023-12-06T15:33:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2023-12-06T15:33:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2023-12-06T15:33:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2023-12-06T15:33:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2023-12-06T15:33:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2023-12-06T15:33:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2023-12-06T15:33:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:33:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1 2023-12-06T15:33:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 2023-12-06T15:33:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster2:1 2023-12-06T15:33:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2023-12-06T15:33:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2023-12-06T15:33:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-12-06T15:33:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster2:2 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:2 2023-12-06T15:33:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:2 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=2 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 2 = cluster2:2 ']' 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=2 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=2 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster2:3 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:3 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:3 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=3 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 3 = cluster2:3 ']' 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=3 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=3 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 2023-12-06T15:33:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.Q1BJHHgE3s/rbd-mirror.test.log 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2023-12-06 15:33:35 TEST: add image and test replay after client crashes 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2023-12-06 15:33:35 TEST: add image and test replay after client crashes 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=test 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:33:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test 512M 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 512M ']' 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=512M 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2023-12-06T15:33:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:33:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-12-06T15:33:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:33:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:33:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:33:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:33:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-12-06T15:33:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:33:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-12-06T15:33:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2023-12-06T15:33:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T15:33:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:33:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10a2d74545b9 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10a2d74545b9 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:33:35 2023 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:33:35 2023 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:33:35 2023 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:33:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:33:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-06T15:33:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:33:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:33:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:33:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-06T15:33:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:33:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:33:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:33:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:33:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:33:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:33:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:33:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-06T15:33:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:33:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:33:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:33:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:33:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:33:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:33:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-06T15:33:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:33:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:33:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:33:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:33:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:33:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:33:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:33:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:33:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:33:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:33:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:33:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:33:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:33:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:33:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:33:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:33:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:33:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:33:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:33:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:33:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:33:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:33:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:33:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-06T15:33:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:2 2023-12-06T15:33:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-06T15:33:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:2 2023-12-06T15:33:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=2 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=2 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=2 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:2 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:2 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=2 2023-12-06T15:33:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-06T15:33:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=2 2023-12-06T15:33:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=2 2023-12-06T15:33:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-06T15:33:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-06T15:33:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:3 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:3 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=3 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-06T15:33:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=3 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=3 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:3 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:3 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=3 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-06T15:33:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=3 2023-12-06T15:33:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=3 2023-12-06T15:33:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-06T15:33:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-06T15:33:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:33:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2023-12-06T15:33:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:33:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:33:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:33:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:33:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:33:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:33:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:33:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:33:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:33:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:33:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:33:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:33:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:33:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-06T15:33:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:2 2023-12-06T15:33:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:2 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=2 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=2 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=2 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-06T15:33:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:2 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:2 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=2 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=2 2023-12-06T15:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=2 2023-12-06T15:33:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-06T15:33:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-06T15:33:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-06T15:33:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:3 2023-12-06T15:33:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:3 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=3 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=3 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=3 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:3 2023-12-06T15:33:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:3 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=3 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=3 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=3 2023-12-06T15:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-06T15:33:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-06T15:33:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:33:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2023-12-06T15:33:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:33:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:33:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:33:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:33:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:33:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:33:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:33:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:33:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:33:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:33:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:33:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:33:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:33:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:33:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:33:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:33:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-06T15:33:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:2 2023-12-06T15:33:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-06T15:33:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:2 2023-12-06T15:33:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=2 2023-12-06T15:33:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-06T15:33:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=2 2023-12-06T15:33:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=2 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:2 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:2 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=2 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=2 2023-12-06T15:33:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=2 2023-12-06T15:33:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-06T15:33:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-06T15:33:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:3 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:3 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=3 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-06T15:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=3 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=3 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:3 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:3 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=3 2023-12-06T15:33:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-06T15:33:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=3 2023-12-06T15:33:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=3 2023-12-06T15:33:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-06T15:33:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-06T15:33:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:33:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 4 2023-12-06T15:33:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:33:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:33:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:33:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:33:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:33:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:33:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:33:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:33:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:33:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:33:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:33:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:33:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:33:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:33:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:33:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:33:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:33:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:33:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:33:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:33:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-06T15:33:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:2 2023-12-06T15:33:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:2 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=2 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=2 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=2 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:2 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-06T15:33:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:2 2023-12-06T15:33:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=2 2023-12-06T15:33:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-06T15:33:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=2 2023-12-06T15:33:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=2 2023-12-06T15:33:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-06T15:33:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-06T15:33:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-06T15:33:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:3 2023-12-06T15:33:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-06T15:33:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:3 2023-12-06T15:33:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=3 2023-12-06T15:33:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-06T15:33:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=3 2023-12-06T15:33:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=3 2023-12-06T15:33:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-06T15:33:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:3 2023-12-06T15:33:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-06T15:33:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:3 2023-12-06T15:33:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=3 2023-12-06T15:33:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-06T15:33:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=3 2023-12-06T15:33:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=3 2023-12-06T15:33:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-06T15:33:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-06T15:33:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:33:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 8 2023-12-06T15:33:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:33:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:33:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:33:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:33:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:33:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:33:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:33:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:33:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:33:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:33:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:33:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:33:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:33:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:33:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:33:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:33:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:33:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:33:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-06T15:33:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:2 2023-12-06T15:33:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:2 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=2 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=2 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=2 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:2 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-06T15:33:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:2 2023-12-06T15:33:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=2 2023-12-06T15:33:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-06T15:33:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=2 2023-12-06T15:33:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=2 2023-12-06T15:33:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-06T15:33:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-06T15:33:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-06T15:33:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:3 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:3 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=3 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=3 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=3 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:3 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:3 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=3 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=3 2023-12-06T15:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=3 2023-12-06T15:33:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-06T15:33:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-06T15:33:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-06T15:33:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:33:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:33:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:33:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:33:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 8 2023-12-06T15:33:57.181 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6852185294109936) > weight (0.25) 2023-12-06T15:33:57.181 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-06T15:33:57.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-06T15:33:57.182 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-06T15:33:57.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi204.stderr:2023-12-06T15:33:57.182+0000 7febe4514700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 119336) UID: 1000 2023-12-06T15:33:57.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:33:58.182 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-06T15:33:58.183 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:33:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:33:58.183 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-06T15:33:58.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-06T15:33:58.183 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-06T15:33:58.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-06T15:33:58.186 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-12-06T15:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:34:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:34:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:34:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:34:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:34:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:34:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:34:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:34:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:34:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:34:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:34:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:34:02.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:34:02.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:34:02.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:34:02.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:34:02.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:34:02.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:34:02.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:34:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:34:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:34:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:34:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:34:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:34:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:34:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:34:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:34:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:34:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:34:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:34:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:34:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:34:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:34:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:34:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:34:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:34:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:34:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:34:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:34:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:34:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:34:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:34:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clean_snap_name= 2023-12-06T15:34:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 10 2023-12-06T15:34:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:34:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror test 2023-12-06T15:34:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:34:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:34:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=7 2023-12-06T15:34:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:34:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-06T15:34:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 45796352 bytes 2023-12-06T15:34:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 82004992 bytes 2023-12-06T15:34:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 100860928 bytes 2023-12-06T15:34:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 102547456 bytes 2023-12-06T15:34:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 148071424 bytes 2023-12-06T15:34:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 236734464 bytes 2023-12-06T15:34:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 246859776 bytes 2023-12-06T15:34:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 291979264 bytes 2023-12-06T15:34:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 358816768 bytes 2023-12-06T15:34:04.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T15:34:04.458+0000 7f6db4cb96c0 -1 rbd::mirror::Mirror: 0x55ccc7cc99e0 update_pool_replayers: restarting failed pool replayer for uuid: 171ac006-ef66-4677-9f7f-b79571e89b7d cluster: cluster2 client: client.mirror 2023-12-06T15:34:04.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T15:34:04.472+0000 7fbf510986c0 -1 rbd::mirror::Mirror: 0x5575bd3bb9e0 update_pool_replayers: restarting failed pool replayer for uuid: 171ac006-ef66-4677-9f7f-b79571e89b7d cluster: cluster2 client: client.mirror 2023-12-06T15:34:04.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:34:04.478+0000 7f7186dc46c0 -1 rbd::mirror::Mirror: 0x5594138a79e0 update_pool_replayers: restarting failed pool replayer for uuid: 171ac006-ef66-4677-9f7f-b79571e89b7d cluster: cluster2 client: client.mirror 2023-12-06T15:34:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 386155520 bytes 2023-12-06T15:34:04.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T15:34:04.566+0000 7f6db4cb96c0 -1 rbd::mirror::Mirror: 0x55ccc7cc99e0 update_pool_replayers: restarting failed pool replayer for uuid: d6f3ff8c-5996-4612-860a-32288d13eb46 cluster: cluster2 client: client.mirror 2023-12-06T15:34:04.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T15:34:04.568+0000 7fbf510986c0 -1 rbd::mirror::Mirror: 0x5575bd3bb9e0 update_pool_replayers: restarting failed pool replayer for uuid: d6f3ff8c-5996-4612-860a-32288d13eb46 cluster: cluster2 client: client.mirror 2023-12-06T15:34:04.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:34:04.592+0000 7f7186dc46c0 -1 rbd::mirror::Mirror: 0x5594138a79e0 update_pool_replayers: restarting failed pool replayer for uuid: d6f3ff8c-5996-4612-860a-32288d13eb46 cluster: cluster2 client: client.mirror 2023-12-06T15:34:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 410673152 bytes 2023-12-06T15:34:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 462848000 bytes 2023-12-06T15:34:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 479115264 bytes 2023-12-06T15:34:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 486373376 bytes 2023-12-06T15:34:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 506727424 bytes 2023-12-06T15:34:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 536812544 bytes 2023-12-06T15:34:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 593978368 bytes 2023-12-06T15:34:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 662464512 bytes 2023-12-06T15:34:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 705060864 bytes 2023-12-06T15:34:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 766372864 bytes 2023-12-06T15:34:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 783169536 bytes 2023-12-06T15:34:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 855348224 bytes 2023-12-06T15:34:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 905505792 bytes 2023-12-06T15:34:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 947345408 bytes 2023-12-06T15:34:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 992328704 bytes 2023-12-06T15:34:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1002355712 bytes 2023-12-06T15:34:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1041426432 bytes 2023-12-06T15:34:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1071167488 bytes 2023-12-06T15:34:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1093231616 bytes 2023-12-06T15:34:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1116649472 bytes 2023-12-06T15:34:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1149235200 bytes 2023-12-06T15:34:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1256909824 bytes 2023-12-06T15:34:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1302278144 bytes 2023-12-06T15:34:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1302887424 bytes 2023-12-06T15:34:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1331476480 bytes 2023-12-06T15:34:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1371482112 bytes 2023-12-06T15:34:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1407779840 bytes 2023-12-06T15:34:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1425484800 bytes 2023-12-06T15:34:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1426108416 bytes 2023-12-06T15:34:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1426703360 bytes 2023-12-06T15:34:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1470281728 bytes 2023-12-06T15:34:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1543155712 bytes 2023-12-06T15:34:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1600229376 bytes 2023-12-06T15:34:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1622500352 bytes 2023-12-06T15:34:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1646203904 bytes 2023-12-06T15:34:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1704837120 bytes 2023-12-06T15:34:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1727320064 bytes 2023-12-06T15:34:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1794114560 bytes 2023-12-06T15:34:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:34:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:34:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:34:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:34:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-06T15:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:34:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:34:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:34:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:34:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:34:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:34:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:34:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:34:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:34:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:34:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:34:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:34:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2023-12-06T15:34:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2023-12-06T15:34:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06T15:34:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:34:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:34:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:34:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:34:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:34:06 2023-12-06T15:34:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2023-12-06T15:34:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:34:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2023-12-06T15:34:11.182 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8964425547067498) > weight (0.25) 2023-12-06T15:34:11.182 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8931028201351112) > weight (0.25) 2023-12-06T15:34:11.182 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-06T15:34:11.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-06T15:34:11.182 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-06T15:34:11.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi204.stderr:2023-12-06T15:34:11.182+0000 7f7222bca700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 119451) UID: 1000 2023-12-06T15:34:11.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:34:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:34:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:34:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:34:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:34:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:34:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:34:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:34:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:34:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:34:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:34:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:34:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2023-12-06T15:34:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2023-12-06T15:34:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06T15:34:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:34:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:34:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:34:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:34:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:34:06 2023-12-06T15:34:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2023-12-06T15:34:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:34:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2023-12-06T15:34:15.183 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-06T15:34:15.183 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:34:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:34:15.184 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-06T15:34:15.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-06T15:34:15.184 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-06T15:34:15.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-06T15:34:15.187 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2023-12-06T15:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:34:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:34:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:34:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:34:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:34:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:34:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:34:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:34:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:34:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2023-12-06T15:34:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2023-12-06T15:34:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06T15:34:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:34:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:34:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:34:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:34:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:34:06 2023-12-06T15:34:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2023-12-06T15:34:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:34:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2023-12-06T15:34:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:34:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:34:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:34:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:34:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:34:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:34:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:34:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:34:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:34:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:34:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:34:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2023-12-06T15:34:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2023-12-06T15:34:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06T15:34:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:34:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:34:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:34:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:34:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:34:06 2023-12-06T15:34:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2023-12-06T15:34:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:34:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2023-12-06T15:34:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:34:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:34:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:34:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:34:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:34:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:34:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:34:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:34:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:34:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2023-12-06T15:34:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701876816,"remote_snapshot_timestamp":1701876816,"replay_state":"idle"} 2023-12-06T15:34:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2023-12-06T15:34:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:34:28 2023-12-06T15:34:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:34:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:34:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:34:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:34:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:34:06 2023-12-06T15:34:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2023-12-06T15:34:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701876816,"remote_snapshot_timestamp":1701876816,"replay_state":"idle"} 2023-12-06T15:34:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:34:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:34:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2023-12-06T15:34:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:34:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2023-12-06T15:34:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'last_update=2023-12-06 15:34:28' 2023-12-06T15:34:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-06 15:34:28' 2023-12-06T15:34:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:34:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2023-12-06 15:34:28' 2023-12-06T15:34:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-06T15:34:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror.mirror_status 2023-12-06T15:34:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:34:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror.mirror_status 2023-12-06T15:34:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-06T15:34:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:34:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2023-12-06T15:34:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-06T15:34:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2023-12-06 15:34:28' 2023-12-06T15:34:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-06T15:34:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2023-12-06T15:34:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:34:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2023-12-06T15:34:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:34:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:34:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:34:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap1 2023-12-06T15:34:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap1 2023-12-06T15:34:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:34:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap1 2023-12-06T15:34:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:34:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-06T15:34:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-06T15:34:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:34:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:34:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:34:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:34:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-06T15:34:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:34:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:34:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:34:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-06T15:34:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:34:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:34:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:34:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:34:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:34:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:34:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:34:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:34:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:34:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:34:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:34:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:34:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:34:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:34:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:34:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:34:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:34:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:34:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:34:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:34:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:34:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:34:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:34:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:34:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:34:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:34:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:34:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:34:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:34:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:34:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:34:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:34:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:34:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:34:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:34:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:34:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:34:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:34:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:34:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:34:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:34:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:34:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:34:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:34:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:34:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:34:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:34:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:34:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:34:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:34:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-06T15:34:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:34:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:34:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:34:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:34:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-06T15:34:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:34:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:34:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.status 2023-12-06T15:34:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.status 2023-12-06T15:34:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:34:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:34:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.status 2023-12-06T15:34:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.status 2023-12-06T15:34:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:34:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:34:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-06T15:34:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:34:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-06T15:34:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 6 2023-12-06T15:34:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:34:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:34:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:34:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-06T15:34:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:34:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:34:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=6 2023-12-06T15:34:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:34:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:34:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:34:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:34:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:34:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:34:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:34:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:34:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=4 2023-12-06T15:34:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 6 = 4 2023-12-06T15:34:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:34:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:34:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:34:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:34:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:34:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:34:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=4 2023-12-06T15:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 6 = 4 2023-12-06T15:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:34:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:34:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:34:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:34:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:34:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:34:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:34:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:34:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:34:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:34:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:34:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 6 = '' 2023-12-06T15:34:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:34:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:34:44.071 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:34:44.070+0000 7f44ae2f8700 -1 reset not still connected to 0x559a05b86b60 2023-12-06T15:34:44.072 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:34:44.070+0000 7f44ae2f8700 -1 reset not still connected to 0x559a0bcda750 2023-12-06T15:34:44.072 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:34:44.070+0000 7f44ae2f8700 -1 reset not still connected to 0x559a0bcdb110 2023-12-06T15:34:44.072 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:34:44.070+0000 7f44ae2f8700 -1 reset not still connected to 0x559a0bcdb520 2023-12-06T15:34:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:34:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:34:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:34:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:34:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:34:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:34:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:34:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:34:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:34:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:34:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 6 = '' 2023-12-06T15:34:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:34:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2023-12-06T15:34:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:34:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:34:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:34:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:34:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:34:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=6 2023-12-06T15:34:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 6 = 6 2023-12-06T15:34:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:34:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-12-06T15:34:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap1 2023-12-06T15:34:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:34:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:34:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:34:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-12-06T15:34:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2023-12-06T15:34:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T15:34:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:34:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:34:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2376d3f3f 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2376d3f3f 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:34:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:34:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:34:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:34:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap1 2023-12-06T15:34:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap1 2023-12-06T15:34:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:34:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:34:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:34:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:34:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:34:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:34:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:34:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:34:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:34:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:34:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:34:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clean_snap_name=snap1-clean 2023-12-06T15:34:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-12-06T15:34:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:34:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap1-clean 2023-12-06T15:34:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-12-06T15:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror test 2023-12-06T15:34:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:34:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:34:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=9 2023-12-06T15:34:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:34:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-06T15:34:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 626688 bytes 2023-12-06T15:34:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 32662528 bytes 2023-12-06T15:34:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 74946560 bytes 2023-12-06T15:34:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 97508352 bytes 2023-12-06T15:34:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 147373056 bytes 2023-12-06T15:34:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 237807616 bytes 2023-12-06T15:34:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 262366208 bytes 2023-12-06T15:34:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 323938304 bytes 2023-12-06T15:34:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 371758080 bytes 2023-12-06T15:34:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 480278528 bytes 2023-12-06T15:34:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 480887808 bytes 2023-12-06T15:34:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 542093312 bytes 2023-12-06T15:34:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 636064768 bytes 2023-12-06T15:34:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 710261760 bytes 2023-12-06T15:34:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 754200576 bytes 2023-12-06T15:34:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 777785344 bytes 2023-12-06T15:34:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 833153024 bytes 2023-12-06T15:34:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 873195520 bytes 2023-12-06T15:34:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 902500352 bytes 2023-12-06T15:34:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 936833024 bytes 2023-12-06T15:34:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 979021824 bytes 2023-12-06T15:34:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 979650560 bytes 2023-12-06T15:34:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1000216576 bytes 2023-12-06T15:34:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1012137984 bytes 2023-12-06T15:34:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1073138688 bytes 2023-12-06T15:34:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1097171968 bytes 2023-12-06T15:34:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1154561024 bytes 2023-12-06T15:34:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1225012224 bytes 2023-12-06T15:34:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1233454080 bytes 2023-12-06T15:34:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1260605440 bytes 2023-12-06T15:34:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1270857728 bytes 2023-12-06T15:34:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1278349312 bytes 2023-12-06T15:34:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1305871360 bytes 2023-12-06T15:34:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1338287104 bytes 2023-12-06T15:34:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1380621312 bytes 2023-12-06T15:34:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1388868608 bytes 2023-12-06T15:34:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1440754688 bytes 2023-12-06T15:34:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1464957952 bytes 2023-12-06T15:34:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1493547008 bytes 2023-12-06T15:34:55.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1531578368 bytes 2023-12-06T15:34:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1586275328 bytes 2023-12-06T15:34:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1613245440 bytes 2023-12-06T15:34:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1629180928 bytes 2023-12-06T15:34:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1686265856 bytes 2023-12-06T15:34:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1719977984 bytes 2023-12-06T15:34:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1745028096 bytes 2023-12-06T15:34:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1805454336 bytes 2023-12-06T15:34:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1842973696 bytes 2023-12-06T15:34:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1884126208 bytes 2023-12-06T15:34:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1922207744 bytes 2023-12-06T15:34:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 1992118272 bytes 2023-12-06T15:34:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2050132992 bytes 2023-12-06T15:34:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2092823552 bytes 2023-12-06T15:34:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2123784192 bytes 2023-12-06T15:34:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2196412416 bytes 2023-12-06T15:34:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2216777728 bytes 2023-12-06T15:34:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2243470336 bytes 2023-12-06T15:34:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2305531904 bytes 2023-12-06T15:34:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:34:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:34:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:34:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:34:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-06T15:34:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:34:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:34:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:34:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:34:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:34:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:34:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:34:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:34:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:34:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:34:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:34:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:34:59.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:34:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:34:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:34:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:34:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2023-12-06T15:34:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":17651985.07,"bytes_per_snapshot":264779776.0,"last_snapshot_bytes":529559552,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701876881,"remote_snapshot_timestamp":1701876881,"replay_state":"idle"} 2023-12-06T15:34:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2023-12-06T15:34:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:34:58 2023-12-06T15:34:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:34:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:34:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:34:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:34:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:34:34 2023-12-06T15:34:59.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2023-12-06T15:34:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":17651985.07,"bytes_per_snapshot":264779776.0,"last_snapshot_bytes":529559552,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701876881,"remote_snapshot_timestamp":1701876881,"replay_state":"idle"} 2023-12-06T15:34:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:34:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:34:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2023-12-06T15:34:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:34:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2023-12-06T15:34:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:34:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'last_update=2023-12-06 15:34:58' 2023-12-06T15:34:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-06 15:34:58' 2023-12-06T15:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2023-12-06 15:34:58' 2023-12-06T15:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-06T15:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror.mirror_status 2023-12-06T15:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:34:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror.mirror_status 2023-12-06T15:34:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-06T15:34:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:34:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2023-12-06T15:34:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-06T15:34:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2023-12-06 15:34:58' 2023-12-06T15:34:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-06T15:34:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2023-12-06T15:34:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:34:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2023-12-06T15:34:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:34:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:34:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:34:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap2 2023-12-06T15:34:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap2 2023-12-06T15:34:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:34:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:34:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:34:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2023-12-06T15:34:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:35:21.188 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6354899691115887) > weight (0.25) 2023-12-06T15:35:21.188 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.31112182380162445) > weight (0.25) 2023-12-06T15:35:21.188 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-06T15:35:21.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-06T15:35:21.189 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-06T15:35:21.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T15:35:21.188+0000 7f6da3a58700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 119359) UID: 1000 2023-12-06T15:35:21.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:35:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-06T15:35:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:35:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-06T15:35:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:35:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:35:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:35:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:35:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:35:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-06T15:35:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:35:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:35:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:35:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:35:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-06T15:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:35:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:35:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:35:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:35:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:35:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:35:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:35:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:35:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:35:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:35:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:35:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:35:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:35:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:35:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:35:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:35:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:35:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:35:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:35:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:35:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:35:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:35:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:35:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:35:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:35:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:35:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:35:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:35:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:35:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:35:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:35:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:35:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:35:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:35:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:35:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:35:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:35:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:35:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:35:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:35:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:35:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:35:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-06T15:35:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.status 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.status 2023-12-06T15:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:35:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:35:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.status 2023-12-06T15:35:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.status 2023-12-06T15:35:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:35:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:35:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-06T15:35:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:35:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-06T15:35:30.189 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-06T15:35:30.190 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:35:30.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:35:30.190 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-06T15:35:30.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-06T15:35:30.190 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-06T15:35:30.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-06T15:35:30.193 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-06T15:35:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 9 2023-12-06T15:35:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:35:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:35:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:35:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-06T15:35:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:35:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:35:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=9 2023-12-06T15:35:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:35:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:35:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:35:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:35:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:35:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:35:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:35:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:35:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:35:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:35:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:35:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:35:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:35:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:35:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 9 = '' 2023-12-06T15:35:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:35:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:35:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:35:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:35:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:35:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:35:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:35:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:35:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:35:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:35:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:35:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:35:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:35:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 9 = '' 2023-12-06T15:35:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:35:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:35:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:35:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:35:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:35:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:35:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:35:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:35:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:35:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:35:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:35:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:35:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:35:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 9 = '' 2023-12-06T15:35:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:35:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:35:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:35:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:35:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:35:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:35:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:35:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:35:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:35:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:35:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:35:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:35:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:35:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 9 = '' 2023-12-06T15:35:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:35:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2023-12-06T15:35:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:35:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:35:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:35:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:35:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:35:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:35:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=9 2023-12-06T15:35:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 9 = 9 2023-12-06T15:35:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:35:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-12-06T15:35:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:35:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2023-12-06T15:35:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:35:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:35:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:35:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-12-06T15:35:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 4 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2376d3f3f 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2376d3f3f 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:35:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:35:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:35:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:35:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:35:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snap1-clean ']' 2023-12-06T15:35:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-06T15:35:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap1-clean 2023-12-06T15:35:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:35:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:35:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:35:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:35:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:35:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:35:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:35:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:35:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap2 2023-12-06T15:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2023-12-06T15:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:35:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:35:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:35:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:35:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:35:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:35:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clean_snap_name=snap2-clean 2023-12-06T15:35:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-12-06T15:35:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:35:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2-clean 2023-12-06T15:35:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-12-06T15:35:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:35:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror test 2023-12-06T15:35:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:35:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:35:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=10 2023-12-06T15:35:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:35:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-06T15:35:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 38055936 bytes 2023-12-06T15:35:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 83704832 bytes 2023-12-06T15:35:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 118697984 bytes 2023-12-06T15:35:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 159532032 bytes 2023-12-06T15:35:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 172317696 bytes 2023-12-06T15:35:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 246493184 bytes 2023-12-06T15:35:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 302277632 bytes 2023-12-06T15:35:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 318955520 bytes 2023-12-06T15:35:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 353398784 bytes 2023-12-06T15:35:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 372753408 bytes 2023-12-06T15:35:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 399622144 bytes 2023-12-06T15:35:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 447059968 bytes 2023-12-06T15:35:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 496618496 bytes 2023-12-06T15:35:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 543572992 bytes 2023-12-06T15:35:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 597043200 bytes 2023-12-06T15:35:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 699760640 bytes 2023-12-06T15:35:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 751749120 bytes 2023-12-06T15:35:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 823912448 bytes 2023-12-06T15:35:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 850222080 bytes 2023-12-06T15:35:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 902656000 bytes 2023-12-06T15:35:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 950418432 bytes 2023-12-06T15:35:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 980374528 bytes 2023-12-06T15:35:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1027565568 bytes 2023-12-06T15:35:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1067043840 bytes 2023-12-06T15:35:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1067663360 bytes 2023-12-06T15:35:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1068274688 bytes 2023-12-06T15:35:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1088662528 bytes 2023-12-06T15:35:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1132267520 bytes 2023-12-06T15:35:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1184703488 bytes 2023-12-06T15:35:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1214068736 bytes 2023-12-06T15:35:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1241234432 bytes 2023-12-06T15:35:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1281509376 bytes 2023-12-06T15:35:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1313695744 bytes 2023-12-06T15:35:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1346290688 bytes 2023-12-06T15:35:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1373433856 bytes 2023-12-06T15:35:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1456632832 bytes 2023-12-06T15:35:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1503289344 bytes 2023-12-06T15:35:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1597382656 bytes 2023-12-06T15:35:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1650197504 bytes 2023-12-06T15:35:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1675008000 bytes 2023-12-06T15:35:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1685457920 bytes 2023-12-06T15:35:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1726040064 bytes 2023-12-06T15:35:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1731022848 bytes 2023-12-06T15:35:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1789427712 bytes 2023-12-06T15:35:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1810674688 bytes 2023-12-06T15:35:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1844664320 bytes 2023-12-06T15:35:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1942543360 bytes 2023-12-06T15:35:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1981805568 bytes 2023-12-06T15:35:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2016337920 bytes 2023-12-06T15:35:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2084560896 bytes 2023-12-06T15:35:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2118153216 bytes 2023-12-06T15:35:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2191152128 bytes 2023-12-06T15:35:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2225912832 bytes 2023-12-06T15:35:50.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2233122816 bytes 2023-12-06T15:35:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2264166400 bytes 2023-12-06T15:35:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2339997696 bytes 2023-12-06T15:35:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2387666944 bytes 2023-12-06T15:35:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2411835392 bytes 2023-12-06T15:35:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2454514688 bytes 2023-12-06T15:35:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2499243008 bytes 2023-12-06T15:35:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2518464512 bytes 2023-12-06T15:35:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2569233408 bytes 2023-12-06T15:35:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2586168320 bytes 2023-12-06T15:35:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2634428416 bytes 2023-12-06T15:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-06T15:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:35:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:35:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:35:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:35:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:35:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:35:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:35:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:35:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:35:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:35:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:35:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:35:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:35:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:35:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:35:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2023-12-06T15:35:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":264779776.0,"last_snapshot_bytes":529559552,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701876881,"remote_snapshot_timestamp":1701876881,"replay_state":"idle"} 2023-12-06T15:35:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2023-12-06T15:35:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:35:28 2023-12-06T15:35:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:35:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:35:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:35:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:35:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:35:34 2023-12-06T15:35:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:35:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2023-12-06T15:35:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:35:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":264779776.0,"last_snapshot_bytes":529559552,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701876881,"remote_snapshot_timestamp":1701876881,"replay_state":"idle"} 2023-12-06T15:35:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:35:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:35:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2023-12-06T15:35:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:35:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:35:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2023-12-06T15:35:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:35:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'last_update=2023-12-06 15:35:28' 2023-12-06T15:35:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-06 15:35:28' 2023-12-06T15:35:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:35:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:35:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2023-12-06 15:35:28' 2023-12-06T15:35:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-06T15:35:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror.mirror_status 2023-12-06T15:35:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:35:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror.mirror_status 2023-12-06T15:35:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-06T15:35:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:35:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2023-12-06T15:35:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-06T15:35:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2023-12-06 15:35:28' 2023-12-06T15:35:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-06T15:35:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2023-12-06T15:35:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:35:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2023-12-06T15:35:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:35:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:35:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:35:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap3 2023-12-06T15:35:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap3 2023-12-06T15:35:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:35:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:35:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:35:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap3 2023-12-06T15:35:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:36:13.188 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7179281535269545) > weight (0.25) 2023-12-06T15:36:13.188 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5651151625241314) > weight (0.25) 2023-12-06T15:36:13.188 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-06T15:36:13.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-06T15:36:13.189 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-06T15:36:13.189 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-06T15:36:13.189 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:36:13.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi204.stderr:2023-12-06T15:36:13.189+0000 7fc6f2a21700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 122498) UID: 1000 2023-12-06T15:36:13.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:36:19.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:36:19.292 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-06T15:36:19.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-06T15:36:19.293 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-06T15:36:19.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-06T15:36:19.296 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-12-06T15:36:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-06T15:36:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:36:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-06T15:36:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:36:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:36:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:36:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:36:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:36:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-06T15:36:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:36:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:36:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:36:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:36:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-06T15:36:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:36:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:36:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:36:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:36:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:36:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:36:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:36:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:36:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:36:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:36:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:36:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:36:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:36:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:36:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:36:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:36:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:36:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:36:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:36:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:36:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:36:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:36:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:36:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:36:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:36:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:36:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:36:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:36:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:36:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:36:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:36:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:36:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:36:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:36:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:36:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:36:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:36:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:36:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:36:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:36:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:36:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:36:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:36:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:36:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:36:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:36:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:36:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:36:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:36:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:36:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:36:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:36:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:36:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:36:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:36:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:36:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:36:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:36:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:36:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:36:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:36:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:36:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:36:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:36:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:36:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:36:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:36:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:36:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:36:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:36:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:36:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:36:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:36:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:36:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:36:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.status 2023-12-06T15:36:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.status 2023-12-06T15:36:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:36:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:36:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.status 2023-12-06T15:36:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.status 2023-12-06T15:36:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:36:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:36:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-06T15:36:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:36:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-06T15:36:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 13 2023-12-06T15:36:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:36:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:36:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:36:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-06T15:36:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:36:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:36:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=13 2023-12-06T15:36:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:36:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:36:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:36:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:36:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:36:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:36:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:36:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:36:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:36:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:36:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:36:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:36:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:36:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:36:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 13 = '' 2023-12-06T15:36:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:36:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:36:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:36:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:36:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:36:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:36:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:36:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:36:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:36:26.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:36:26.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:36:26.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:36:26.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:36:26.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 13 = '' 2023-12-06T15:36:26.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:36:26.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:36:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:36:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:36:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:36:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:36:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:36:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:36:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:36:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:36:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:36:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:36:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:36:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 13 = '' 2023-12-06T15:36:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:36:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:36:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:36:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:36:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:36:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:36:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:36:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:36:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:36:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:36:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:36:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:36:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:36:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 13 = '' 2023-12-06T15:36:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:36:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2023-12-06T15:36:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:36:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:36:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:36:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:36:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:36:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:36:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=13 2023-12-06T15:36:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 13 = 13 2023-12-06T15:36:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:36:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-12-06T15:36:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:36:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap3 2023-12-06T15:36:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:36:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:36:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:36:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-12-06T15:36:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2023-12-06T15:36:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T15:36:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:36:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 6 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2376d3f3f 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2376d3f3f 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:35:38 2023 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:36:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:36:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:36:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snap2-clean ']' 2023-12-06T15:36:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-06T15:36:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2-clean 2023-12-06T15:36:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:36:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:36:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:36:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:36:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:36:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:36:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:36:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:36:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:36:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:36:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap3 2023-12-06T15:36:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap3 2023-12-06T15:36:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:36:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:36:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:36:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:36:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:36:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:36:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:36:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:36:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:36:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:36:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:36:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clean_snap_name=snap3-clean 2023-12-06T15:36:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-12-06T15:36:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:36:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap3-clean 2023-12-06T15:36:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-12-06T15:36:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:36:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror test 2023-12-06T15:36:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:36:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:36:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=12 2023-12-06T15:36:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:36:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-06T15:36:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 15728640 bytes 2023-12-06T15:36:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 49220608 bytes 2023-12-06T15:36:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 80274432 bytes 2023-12-06T15:36:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 149190656 bytes 2023-12-06T15:36:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 211949568 bytes 2023-12-06T15:36:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 233227264 bytes 2023-12-06T15:36:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 310509568 bytes 2023-12-06T15:36:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 362576896 bytes 2023-12-06T15:36:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 400862208 bytes 2023-12-06T15:36:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 414449664 bytes 2023-12-06T15:36:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 428482560 bytes 2023-12-06T15:36:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 477541376 bytes 2023-12-06T15:36:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 539045888 bytes 2023-12-06T15:36:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 569062400 bytes 2023-12-06T15:36:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 607415296 bytes 2023-12-06T15:36:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 659809280 bytes 2023-12-06T15:36:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 662342656 bytes 2023-12-06T15:36:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 766101504 bytes 2023-12-06T15:36:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 824726528 bytes 2023-12-06T15:36:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 873920512 bytes 2023-12-06T15:36:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 924311552 bytes 2023-12-06T15:36:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 998643712 bytes 2023-12-06T15:36:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1064429568 bytes 2023-12-06T15:36:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1086626816 bytes 2023-12-06T15:36:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1155765248 bytes 2023-12-06T15:36:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1206130688 bytes 2023-12-06T15:36:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1236791296 bytes 2023-12-06T15:36:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1345299456 bytes 2023-12-06T15:36:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1366412288 bytes 2023-12-06T15:36:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1390602240 bytes 2023-12-06T15:36:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1407008768 bytes 2023-12-06T15:36:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1429861376 bytes 2023-12-06T15:36:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1478253568 bytes 2023-12-06T15:36:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1482759168 bytes 2023-12-06T15:36:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1505033216 bytes 2023-12-06T15:36:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1546706944 bytes 2023-12-06T15:36:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1592641536 bytes 2023-12-06T15:36:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1625882624 bytes 2023-12-06T15:36:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1628153856 bytes 2023-12-06T15:36:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1640812544 bytes 2023-12-06T15:36:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1666423808 bytes 2023-12-06T15:36:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1674777600 bytes 2023-12-06T15:36:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1691344896 bytes 2023-12-06T15:36:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1727857664 bytes 2023-12-06T15:36:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1754060800 bytes 2023-12-06T15:36:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1761717248 bytes 2023-12-06T15:36:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1777388544 bytes 2023-12-06T15:36:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1863593984 bytes 2023-12-06T15:36:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1886018560 bytes 2023-12-06T15:36:43.194 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-06T15:36:43.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-06T15:36:43.195 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-06T15:36:43.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:36:43.194+0000 7f7175b63700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 119320) UID: 1000 2023-12-06T15:36:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1937705984 bytes 2023-12-06T15:36:43.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:36:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 1980941312 bytes 2023-12-06T15:36:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2011769856 bytes 2023-12-06T15:36:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2081855488 bytes 2023-12-06T15:36:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2141170688 bytes 2023-12-06T15:36:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2184327168 bytes 2023-12-06T15:36:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2202371072 bytes 2023-12-06T15:36:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2242119680 bytes 2023-12-06T15:36:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2280283136 bytes 2023-12-06T15:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2342452224 bytes 2023-12-06T15:36:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2387158016 bytes 2023-12-06T15:36:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2409427968 bytes 2023-12-06T15:36:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2453691392 bytes 2023-12-06T15:36:45.196 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-06T15:36:45.196 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:36:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:36:45.197 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-06T15:36:45.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-06T15:36:45.197 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-06T15:36:45.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-06T15:36:45.200 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-12-06T15:36:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2483969024 bytes 2023-12-06T15:36:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2506986496 bytes 2023-12-06T15:36:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 2546461696 bytes 2023-12-06T15:36:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6600 IOs, 2595520512 bytes 2023-12-06T15:36:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6700 IOs, 2638031872 bytes 2023-12-06T15:36:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6800 IOs, 2674855936 bytes 2023-12-06T15:36:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6900 IOs, 2768411648 bytes 2023-12-06T15:36:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7000 IOs, 2785572864 bytes 2023-12-06T15:36:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7100 IOs, 2891066368 bytes 2023-12-06T15:36:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7200 IOs, 2938114048 bytes 2023-12-06T15:36:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7300 IOs, 2987231232 bytes 2023-12-06T15:36:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7400 IOs, 3016597504 bytes 2023-12-06T15:36:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7500 IOs, 3017192448 bytes 2023-12-06T15:36:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7600 IOs, 3036726272 bytes 2023-12-06T15:36:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7700 IOs, 3052500992 bytes 2023-12-06T15:36:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:36:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:36:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:36:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:36:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-06T15:36:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:36:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:36:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:36:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:36:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:36:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:36:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:36:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:36:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:36:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:36:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:36:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:36:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:36:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:36:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:36:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:36:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:36:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:36:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:36:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:36:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2023-12-06T15:36:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531176448.0,"last_snapshot_bytes":532793344,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701876930,"remote_snapshot_timestamp":1701876930,"replay_state":"idle"} 2023-12-06T15:36:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2023-12-06T15:36:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:36:28 2023-12-06T15:36:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:36:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:36:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:36:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:36:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:36:34 2023-12-06T15:36:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:36:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2023-12-06T15:36:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:36:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531176448.0,"last_snapshot_bytes":532793344,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701876930,"remote_snapshot_timestamp":1701876930,"replay_state":"idle"} 2023-12-06T15:36:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:36:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:36:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2023-12-06T15:36:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:36:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:36:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2023-12-06T15:36:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:36:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'last_update=2023-12-06 15:36:28' 2023-12-06T15:36:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-06 15:36:28' 2023-12-06T15:36:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:36:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:36:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2023-12-06 15:36:28' 2023-12-06T15:36:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-06T15:36:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror.mirror_status 2023-12-06T15:36:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:36:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror.mirror_status 2023-12-06T15:36:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-06T15:36:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:36:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2023-12-06T15:36:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-06T15:36:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2023-12-06 15:36:28' 2023-12-06T15:36:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-06T15:36:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2023-12-06T15:36:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:36:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2023-12-06T15:36:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:36:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:36:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:36:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap4 2023-12-06T15:36:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap4 2023-12-06T15:36:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:36:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:36:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:36:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap4 2023-12-06T15:36:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:37:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-06T15:37:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:37:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-06T15:37:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:37:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:37:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:37:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:37:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:37:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-06T15:37:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:37:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:37:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:37:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:37:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-06T15:37:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:37:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:37:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:37:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:37:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:37:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:37:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:37:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:37:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:37:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:37:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:37:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:37:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:37:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:37:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:37:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:37:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:37:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:37:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:37:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:37:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:37:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:37:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:37:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:37:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:37:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:37:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:37:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:37:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:37:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:37:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:37:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:37:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:37:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:37:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:37:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:37:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:37:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:37:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:37:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:37:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:37:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:37:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:37:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:37:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:37:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:37:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:37:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:37:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:37:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-06T15:37:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:37:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:37:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:37:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:37:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-06T15:37:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:37:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:37:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.status 2023-12-06T15:37:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.status 2023-12-06T15:37:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:37:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:37:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.status 2023-12-06T15:37:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.status 2023-12-06T15:37:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:37:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:37:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-06T15:37:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:37:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-06T15:37:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 17 2023-12-06T15:37:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:37:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:37:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:37:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-06T15:37:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:37:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:37:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=17 2023-12-06T15:37:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:37:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:37:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:37:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:37:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:37:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:37:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:37:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:37:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:37:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:37:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:37:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:37:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:37:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:37:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 17 = '' 2023-12-06T15:37:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:37:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:37:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:37:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:37:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:37:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:37:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:37:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:37:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:37:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:37:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:37:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:37:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:37:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 17 = '' 2023-12-06T15:37:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:37:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:37:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:37:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:37:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:37:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:37:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:37:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:37:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:37:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:37:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:37:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:37:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:37:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 17 = '' 2023-12-06T15:37:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:37:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:37:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:37:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:37:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:37:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:37:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:37:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:37:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=17 2023-12-06T15:37:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 17 = 17 2023-12-06T15:37:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:37:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-12-06T15:37:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:37:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap4 2023-12-06T15:37:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:37:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:37:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:37:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-12-06T15:37:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2023-12-06T15:37:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T15:37:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:37:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 8 2023-12-06T15:37:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2376d3f3f 2023-12-06T15:37:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2376d3f3f 2023-12-06T15:37:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:37:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:37:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:37:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:37:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:37:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:35:38 2023 2023-12-06T15:37:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:37:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:37:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:37:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:37:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:37:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:37:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:37:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snap3-clean ']' 2023-12-06T15:37:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-06T15:37:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap3-clean 2023-12-06T15:37:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:37:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:37:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:37:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:37:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:37:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:37:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:37:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:37:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:37:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:37:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:37:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap4 2023-12-06T15:37:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap4 2023-12-06T15:37:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:37:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:37:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:37:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:37:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:37:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:37:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:37:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:37:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:37:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:37:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:37:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clean_snap_name=snap4-clean 2023-12-06T15:37:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-12-06T15:37:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:37:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap4-clean 2023-12-06T15:37:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-12-06T15:37:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:37:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror test 2023-12-06T15:37:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:37:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:37:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=14 2023-12-06T15:37:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:37:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-06T15:37:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 36425728 bytes 2023-12-06T15:37:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 67172352 bytes 2023-12-06T15:37:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 90810368 bytes 2023-12-06T15:37:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 115837952 bytes 2023-12-06T15:37:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 163322880 bytes 2023-12-06T15:37:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 217570304 bytes 2023-12-06T15:37:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 259494912 bytes 2023-12-06T15:37:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 309597184 bytes 2023-12-06T15:37:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 323127296 bytes 2023-12-06T15:37:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 371540992 bytes 2023-12-06T15:37:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 408545280 bytes 2023-12-06T15:37:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 467769344 bytes 2023-12-06T15:37:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 486638592 bytes 2023-12-06T15:37:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 554919936 bytes 2023-12-06T15:37:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 611053568 bytes 2023-12-06T15:37:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 696177664 bytes 2023-12-06T15:37:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 712505344 bytes 2023-12-06T15:37:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 731808768 bytes 2023-12-06T15:37:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 771584000 bytes 2023-12-06T15:37:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 827590656 bytes 2023-12-06T15:37:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 861601792 bytes 2023-12-06T15:37:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 949755904 bytes 2023-12-06T15:37:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 981384192 bytes 2023-12-06T15:37:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1040816128 bytes 2023-12-06T15:37:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1097818112 bytes 2023-12-06T15:37:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1164473344 bytes 2023-12-06T15:37:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1171740672 bytes 2023-12-06T15:37:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1180524544 bytes 2023-12-06T15:37:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1310542848 bytes 2023-12-06T15:37:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1361364992 bytes 2023-12-06T15:37:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1467243520 bytes 2023-12-06T15:37:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1523258368 bytes 2023-12-06T15:37:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1539510272 bytes 2023-12-06T15:37:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1556986880 bytes 2023-12-06T15:37:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1643121664 bytes 2023-12-06T15:37:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1685756928 bytes 2023-12-06T15:37:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1707232256 bytes 2023-12-06T15:37:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1795229696 bytes 2023-12-06T15:37:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1808517120 bytes 2023-12-06T15:37:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1828722688 bytes 2023-12-06T15:37:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1881349120 bytes 2023-12-06T15:37:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1903269888 bytes 2023-12-06T15:37:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1991490560 bytes 2023-12-06T15:37:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 2045328384 bytes 2023-12-06T15:37:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 2085113856 bytes 2023-12-06T15:37:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 2137304064 bytes 2023-12-06T15:37:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2220975104 bytes 2023-12-06T15:37:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2347677696 bytes 2023-12-06T15:37:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2352208896 bytes 2023-12-06T15:37:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2392159232 bytes 2023-12-06T15:37:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2403855360 bytes 2023-12-06T15:37:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2442956800 bytes 2023-12-06T15:37:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2488791040 bytes 2023-12-06T15:37:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2546709504 bytes 2023-12-06T15:37:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2562865152 bytes 2023-12-06T15:37:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2647990272 bytes 2023-12-06T15:37:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2677797888 bytes 2023-12-06T15:37:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2714954752 bytes 2023-12-06T15:37:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2742078464 bytes 2023-12-06T15:37:39.591 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2023-12-06T15:37:39.589+0000 7f6458ca5700 -1 reset not still connected to 0x5594edf1f040 2023-12-06T15:37:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2803570688 bytes 2023-12-06T15:37:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2844056576 bytes 2023-12-06T15:37:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2902281216 bytes 2023-12-06T15:37:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2961086464 bytes 2023-12-06T15:37:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2988770304 bytes 2023-12-06T15:37:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 3041860608 bytes 2023-12-06T15:37:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6600 IOs, 3056789504 bytes 2023-12-06T15:37:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6700 IOs, 3080163328 bytes 2023-12-06T15:37:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6800 IOs, 3105911808 bytes 2023-12-06T15:37:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6900 IOs, 3160124416 bytes 2023-12-06T15:37:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7000 IOs, 3207680000 bytes 2023-12-06T15:37:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7100 IOs, 3232378880 bytes 2023-12-06T15:37:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7200 IOs, 3266679808 bytes 2023-12-06T15:37:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7300 IOs, 3289526272 bytes 2023-12-06T15:37:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7400 IOs, 3361031168 bytes 2023-12-06T15:37:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7500 IOs, 3418471424 bytes 2023-12-06T15:37:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7600 IOs, 3435505664 bytes 2023-12-06T15:37:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7700 IOs, 3436107776 bytes 2023-12-06T15:37:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7800 IOs, 3449860096 bytes 2023-12-06T15:37:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7900 IOs, 3476950016 bytes 2023-12-06T15:37:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8000 IOs, 3547462656 bytes 2023-12-06T15:37:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8100 IOs, 3599345664 bytes 2023-12-06T15:37:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8200 IOs, 3615459328 bytes 2023-12-06T15:37:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8300 IOs, 3663087616 bytes 2023-12-06T15:37:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8400 IOs, 3663689728 bytes 2023-12-06T15:37:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:37:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:37:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:37:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:37:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-06T15:37:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:37:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:37:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:37:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:37:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:37:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:37:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:37:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:37:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:37:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:37:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:37:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:37:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2023-12-06T15:37:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":17869073.07,"bytes_per_snapshot":530798080.0,"last_snapshot_bytes":536072192,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701877039,"remote_snapshot_timestamp":1701877039,"replay_state":"idle"} 2023-12-06T15:37:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2023-12-06T15:37:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:37:28 2023-12-06T15:37:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:37:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:37:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:37:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:37:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:37:34 2023-12-06T15:37:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:37:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2023-12-06T15:37:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:37:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":17869073.07,"bytes_per_snapshot":530798080.0,"last_snapshot_bytes":536072192,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701877039,"remote_snapshot_timestamp":1701877039,"replay_state":"idle"} 2023-12-06T15:37:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:37:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:37:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2023-12-06T15:37:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:37:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:37:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2023-12-06T15:37:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:37:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'last_update=2023-12-06 15:37:28' 2023-12-06T15:37:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-06 15:37:28' 2023-12-06T15:37:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:37:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:37:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2023-12-06 15:37:28' 2023-12-06T15:37:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-06T15:37:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror.mirror_status 2023-12-06T15:37:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:37:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror.mirror_status 2023-12-06T15:37:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-06T15:37:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:37:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2023-12-06T15:37:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-06T15:37:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2023-12-06 15:37:28' 2023-12-06T15:37:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-06T15:37:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2023-12-06T15:37:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:37:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2023-12-06T15:37:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:37:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:37:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:37:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap5 2023-12-06T15:37:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap5 2023-12-06T15:37:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:37:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:37:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:37:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap5 2023-12-06T15:37:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:37:49.297 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.47099801227769866) > weight (0.25) 2023-12-06T15:37:49.297 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.33687706448075394) > weight (0.25) 2023-12-06T15:37:49.297 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.766085040339649) > weight (0.25) 2023-12-06T15:37:49.297 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.40808504090623865) > weight (0.25) 2023-12-06T15:37:49.297 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2023-12-06T15:38:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-06T15:38:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:38:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-06T15:38:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:38:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:38:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:38:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:38:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:38:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-06T15:38:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:38:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:38:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:38:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:38:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-06T15:38:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:38:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:38:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:38:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:38:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:38:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:38:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:38:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:38:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:38:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:38:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:38:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:38:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:38:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:38:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:38:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:38:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:38:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:38:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:38:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:38:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:38:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:38:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:38:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:38:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:38:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:38:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:38:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:38:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:38:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:38:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:38:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:38:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:38:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:38:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:38:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:38:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:38:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:38:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:38:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:38:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:38:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:38:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:38:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:38:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:38:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:38:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:38:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:38:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:38:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:38:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:38:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:38:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:38:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:38:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:38:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:38:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:38:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:38:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:38:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:38:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:38:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:38:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:38:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:38:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:38:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-06T15:38:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.status 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.status 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:38:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:38:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.status 2023-12-06T15:38:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.status 2023-12-06T15:38:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:38:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:38:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-06T15:38:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:38:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-06T15:38:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 21 2023-12-06T15:38:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:38:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:38:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:38:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-06T15:38:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:38:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:38:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=21 2023-12-06T15:38:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:38:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:38:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:38:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:38:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:38:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:38:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:38:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:38:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:38:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 21 = '' 2023-12-06T15:38:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:38:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:38:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:38:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:38:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:38:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:38:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:38:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:38:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:38:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:38:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:38:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:38:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:38:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 21 = '' 2023-12-06T15:38:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:38:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:38:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:38:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:38:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:38:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:38:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:38:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:38:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:38:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:38:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:38:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 21 = '' 2023-12-06T15:38:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:38:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:38:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:38:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:38:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:38:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:38:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:38:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:38:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=21 2023-12-06T15:38:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 21 = 21 2023-12-06T15:38:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:38:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-12-06T15:38:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:38:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap5 2023-12-06T15:38:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:38:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:38:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:38:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-12-06T15:38:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2023-12-06T15:38:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T15:38:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:38:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 10 2023-12-06T15:38:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2376d3f3f 2023-12-06T15:38:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2376d3f3f 2023-12-06T15:38:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:38:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:38:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:38:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:38:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:38:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:37:25 2023 2023-12-06T15:38:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:38:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:38:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:38:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:38:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:38:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:38:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:38:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snap4-clean ']' 2023-12-06T15:38:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-06T15:38:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap4-clean 2023-12-06T15:38:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:38:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:38:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:38:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:38:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:38:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:38:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:38:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:38:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:38:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:38:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:38:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap5 2023-12-06T15:38:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap5 2023-12-06T15:38:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:38:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:38:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:38:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:38:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:38:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:38:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:38:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:38:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:38:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:38:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:38:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clean_snap_name=snap5-clean 2023-12-06T15:38:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-12-06T15:38:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:38:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap5-clean 2023-12-06T15:38:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-12-06T15:38:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:38:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror test 2023-12-06T15:38:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:38:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:38:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=13 2023-12-06T15:38:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:38:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-06T15:38:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 11214848 bytes 2023-12-06T15:38:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 68167680 bytes 2023-12-06T15:38:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 129803264 bytes 2023-12-06T15:38:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 181019648 bytes 2023-12-06T15:38:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 260007936 bytes 2023-12-06T15:38:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 280470528 bytes 2023-12-06T15:38:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 289521664 bytes 2023-12-06T15:38:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 327761920 bytes 2023-12-06T15:38:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 379418624 bytes 2023-12-06T15:38:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 397650944 bytes 2023-12-06T15:38:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 446678016 bytes 2023-12-06T15:38:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 477040640 bytes 2023-12-06T15:38:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 480412672 bytes 2023-12-06T15:38:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 541249536 bytes 2023-12-06T15:38:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 585609216 bytes 2023-12-06T15:38:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 615214080 bytes 2023-12-06T15:38:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 622266368 bytes 2023-12-06T15:38:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 655233024 bytes 2023-12-06T15:38:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 722763776 bytes 2023-12-06T15:38:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 761763840 bytes 2023-12-06T15:38:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 821423104 bytes 2023-12-06T15:38:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 844637184 bytes 2023-12-06T15:38:26.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 897822720 bytes 2023-12-06T15:38:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 954143744 bytes 2023-12-06T15:38:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 973154304 bytes 2023-12-06T15:38:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1014114304 bytes 2023-12-06T15:38:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1025914880 bytes 2023-12-06T15:38:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1070342144 bytes 2023-12-06T15:38:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1079441408 bytes 2023-12-06T15:38:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1086726144 bytes 2023-12-06T15:38:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1106093056 bytes 2023-12-06T15:38:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1132723200 bytes 2023-12-06T15:38:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1171445760 bytes 2023-12-06T15:38:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1220628480 bytes 2023-12-06T15:38:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1296456704 bytes 2023-12-06T15:38:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1318786048 bytes 2023-12-06T15:38:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1337305088 bytes 2023-12-06T15:38:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1338735616 bytes 2023-12-06T15:38:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1419360256 bytes 2023-12-06T15:38:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1522427904 bytes 2023-12-06T15:38:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1574455296 bytes 2023-12-06T15:38:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1664048128 bytes 2023-12-06T15:38:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1724196864 bytes 2023-12-06T15:38:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1735773184 bytes 2023-12-06T15:38:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1769277440 bytes 2023-12-06T15:38:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1802763264 bytes 2023-12-06T15:38:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1855518720 bytes 2023-12-06T15:38:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1884310528 bytes 2023-12-06T15:38:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1939582976 bytes 2023-12-06T15:38:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2012420096 bytes 2023-12-06T15:38:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2114095104 bytes 2023-12-06T15:38:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2133974016 bytes 2023-12-06T15:38:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2134788096 bytes 2023-12-06T15:38:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2191981568 bytes 2023-12-06T15:38:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2279149568 bytes 2023-12-06T15:38:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2370532352 bytes 2023-12-06T15:38:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2417270784 bytes 2023-12-06T15:38:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2493469696 bytes 2023-12-06T15:38:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2545327104 bytes 2023-12-06T15:38:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2581673984 bytes 2023-12-06T15:38:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2639315968 bytes 2023-12-06T15:38:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2671761408 bytes 2023-12-06T15:38:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2695135232 bytes 2023-12-06T15:38:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2746157056 bytes 2023-12-06T15:38:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 2752035840 bytes 2023-12-06T15:38:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6600 IOs, 2799760384 bytes 2023-12-06T15:38:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6700 IOs, 2813331456 bytes 2023-12-06T15:38:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6800 IOs, 2819402752 bytes 2023-12-06T15:38:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6900 IOs, 2884250624 bytes 2023-12-06T15:38:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7000 IOs, 2900140032 bytes 2023-12-06T15:38:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7100 IOs, 2927425536 bytes 2023-12-06T15:38:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7200 IOs, 2951378944 bytes 2023-12-06T15:38:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7300 IOs, 2985141248 bytes 2023-12-06T15:38:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7400 IOs, 3035450368 bytes 2023-12-06T15:38:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7500 IOs, 3072359424 bytes 2023-12-06T15:38:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7600 IOs, 3110634496 bytes 2023-12-06T15:38:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7700 IOs, 3161223168 bytes 2023-12-06T15:38:35.201 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-06T15:38:35.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-06T15:38:35.201 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-06T15:38:35.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:38:35.201+0000 7ff74c7c8700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 124343) UID: 1000 2023-12-06T15:38:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7800 IOs, 3229160448 bytes 2023-12-06T15:38:35.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:38:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7900 IOs, 3257679872 bytes 2023-12-06T15:38:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8000 IOs, 3281949696 bytes 2023-12-06T15:38:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:38:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:38:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:38:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:38:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-06T15:38:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:38:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:38:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:38:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:38:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:38:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:38:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:38:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:38:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:38:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:38:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:38:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:38:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:38:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:38:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:38:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:38:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:38:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:38:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:38:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:38:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2023-12-06T15:38:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":17819272.53,"bytes_per_snapshot":535325184.0,"last_snapshot_bytes":534578176,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701877093,"remote_snapshot_timestamp":1701877093,"replay_state":"idle"} 2023-12-06T15:38:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2023-12-06T15:38:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:38:28 2023-12-06T15:38:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:38:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:38:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:38:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:38:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:38:34 2023-12-06T15:38:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:38:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2023-12-06T15:38:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:38:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":17819272.53,"bytes_per_snapshot":535325184.0,"last_snapshot_bytes":534578176,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701877093,"remote_snapshot_timestamp":1701877093,"replay_state":"idle"} 2023-12-06T15:38:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:38:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:38:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2023-12-06T15:38:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:38:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:38:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2023-12-06T15:38:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:38:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'last_update=2023-12-06 15:38:28' 2023-12-06T15:38:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-06 15:38:28' 2023-12-06T15:38:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:38:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:38:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2023-12-06 15:38:28' 2023-12-06T15:38:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-06T15:38:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror.mirror_status 2023-12-06T15:38:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:38:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror.mirror_status 2023-12-06T15:38:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-06T15:38:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:38:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2023-12-06T15:38:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-06T15:38:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2023-12-06 15:38:28' 2023-12-06T15:38:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-06T15:38:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2023-12-06T15:38:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:38:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2023-12-06T15:38:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:38:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:38:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:38:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap6 2023-12-06T15:38:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap6 2023-12-06T15:38:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:38:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:38:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:38:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap6 2023-12-06T15:38:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:38:37.202 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-06T15:38:37.202 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:38:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:38:37.203 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-06T15:38:37.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-06T15:38:37.204 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-06T15:38:37.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-06T15:38:37.207 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2023-12-06T15:38:59.298 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8059839562059953) > weight (0.25) 2023-12-06T15:38:59.299 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5149951859323604) > weight (0.25) 2023-12-06T15:38:59.299 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-06T15:38:59.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-06T15:38:59.299 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-06T15:38:59.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi204.stderr:2023-12-06T15:38:59.299+0000 7fe9cae7a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 123831) UID: 1000 2023-12-06T15:38:59.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:39:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-06T15:39:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-06T15:39:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:39:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:39:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:39:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:39:08.300 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-06T15:39:08.301 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:39:08.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:39:08.301 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-06T15:39:08.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-06T15:39:08.302 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-06T15:39:08.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-06T15:39:08.305 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-12-06T15:39:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-06T15:39:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:39:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:39:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:39:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-06T15:39:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:39:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:39:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:39:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:39:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:39:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:39:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:39:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:39:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:39:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:39:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:39:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:39:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:39:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:39:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:39:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:39:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:39:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:39:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:39:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:39:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:39:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:39:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:39:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:39:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:39:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:39:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:39:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:39:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:39:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:39:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:39:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:39:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:39:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:39:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:39:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:39:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:39:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:39:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:39:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:39:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:39:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:39:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:39:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:39:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:39:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:39:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:39:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:39:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:39:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:39:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:39:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:39:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:39:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:39:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-06T15:39:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.status 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.status 2023-12-06T15:39:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:39:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:39:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.status 2023-12-06T15:39:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.status 2023-12-06T15:39:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:39:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:39:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-06T15:39:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:39:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-06T15:39:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 25 2023-12-06T15:39:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:39:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:39:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:39:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-06T15:39:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:39:10.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:39:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=25 2023-12-06T15:39:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:39:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:39:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:39:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:39:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:39:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:39:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:39:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:39:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:39:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:39:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:39:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:39:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:39:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 25 = '' 2023-12-06T15:39:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:39:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:39:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:39:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:39:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:39:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:39:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:39:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:39:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:39:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:39:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:39:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:39:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 25 = '' 2023-12-06T15:39:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:39:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:39:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:39:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:39:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:39:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:39:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:39:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:39:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:39:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:39:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:39:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:39:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 25 = '' 2023-12-06T15:39:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:39:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:39:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:39:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:39:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:39:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:39:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:39:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:39:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:39:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:39:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:39:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 25 = '' 2023-12-06T15:39:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:39:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2023-12-06T15:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:39:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:39:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=25 2023-12-06T15:39:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 25 = 25 2023-12-06T15:39:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:39:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-12-06T15:39:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap6 2023-12-06T15:39:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:39:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:39:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:39:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-12-06T15:39:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2023-12-06T15:39:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 12 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2376d3f3f 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2376d3f3f 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:37:25 2023 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:39:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:39:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:39:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snap5-clean ']' 2023-12-06T15:39:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-06T15:39:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap5-clean 2023-12-06T15:39:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:39:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:39:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:39:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:39:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:39:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:39:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:39:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:39:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:39:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:39:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap6 2023-12-06T15:39:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap6 2023-12-06T15:39:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:39:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:39:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:39:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:39:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:39:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:39:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:39:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:39:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:39:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:39:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:39:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clean_snap_name=snap6-clean 2023-12-06T15:39:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-12-06T15:39:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:39:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap6-clean 2023-12-06T15:39:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-12-06T15:39:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:39:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror test 2023-12-06T15:39:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:39:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:39:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=6 2023-12-06T15:39:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:39:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-06T15:39:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 629760 bytes 2023-12-06T15:39:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 40818688 bytes 2023-12-06T15:39:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 76099584 bytes 2023-12-06T15:39:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 160490496 bytes 2023-12-06T15:39:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 229244928 bytes 2023-12-06T15:39:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 276235264 bytes 2023-12-06T15:39:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 308913152 bytes 2023-12-06T15:39:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 368830464 bytes 2023-12-06T15:39:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 423770112 bytes 2023-12-06T15:39:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 527648768 bytes 2023-12-06T15:39:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 588631040 bytes 2023-12-06T15:39:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 590301184 bytes 2023-12-06T15:39:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 655567872 bytes 2023-12-06T15:39:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 669956096 bytes 2023-12-06T15:39:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 686926848 bytes 2023-12-06T15:39:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 703400960 bytes 2023-12-06T15:39:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 764122112 bytes 2023-12-06T15:39:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 768210944 bytes 2023-12-06T15:39:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 792480768 bytes 2023-12-06T15:39:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 836391936 bytes 2023-12-06T15:39:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 897734656 bytes 2023-12-06T15:39:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 912763904 bytes 2023-12-06T15:39:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 948341760 bytes 2023-12-06T15:39:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 974865408 bytes 2023-12-06T15:39:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1055590400 bytes 2023-12-06T15:39:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1104506880 bytes 2023-12-06T15:39:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1144931328 bytes 2023-12-06T15:39:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1182852096 bytes 2023-12-06T15:39:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1251966976 bytes 2023-12-06T15:39:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1335666688 bytes 2023-12-06T15:39:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1342688256 bytes 2023-12-06T15:39:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1358629888 bytes 2023-12-06T15:39:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1413258240 bytes 2023-12-06T15:39:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1465353216 bytes 2023-12-06T15:39:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:39:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:39:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:39:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:39:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-06T15:39:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:39:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:39:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:39:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:39:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:39:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:39:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:39:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:39:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:39:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:39:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:39:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:39:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:39:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:39:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:39:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:39:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2023-12-06T15:39:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535325184.0,"last_snapshot_bytes":534578176,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701877093,"remote_snapshot_timestamp":1701877093,"replay_state":"idle"} 2023-12-06T15:39:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2023-12-06T15:39:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:38:58 2023-12-06T15:39:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:39:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:39:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:39:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:39:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:39:04 2023-12-06T15:39:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:39:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2023-12-06T15:39:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:39:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535325184.0,"last_snapshot_bytes":534578176,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701877093,"remote_snapshot_timestamp":1701877093,"replay_state":"idle"} 2023-12-06T15:39:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:39:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:39:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2023-12-06T15:39:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:39:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:39:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2023-12-06T15:39:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:39:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'last_update=2023-12-06 15:38:58' 2023-12-06T15:39:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-06 15:38:58' 2023-12-06T15:39:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:39:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2023-12-06 15:38:58' 2023-12-06T15:39:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-06T15:39:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror.mirror_status 2023-12-06T15:39:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:39:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror.mirror_status 2023-12-06T15:39:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-06T15:39:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:39:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2023-12-06T15:39:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-06T15:39:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2023-12-06 15:38:58' 2023-12-06T15:39:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-06T15:39:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2023-12-06T15:39:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:39:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2023-12-06T15:39:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:39:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:39:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:39:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap7 2023-12-06T15:39:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap7 2023-12-06T15:39:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:39:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap7 2023-12-06T15:39:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:39:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-06T15:39:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-06T15:39:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:39:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:39:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:39:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:39:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-06T15:39:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:39:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:39:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:39:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:39:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-06T15:39:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:39:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:39:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:39:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:39:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:39:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:39:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:39:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:39:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:39:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:39:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:39:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:39:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:39:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:39:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:39:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:39:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:39:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:39:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:39:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:39:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:39:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:39:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:39:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:39:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:39:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:39:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:39:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:39:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:39:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:39:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:39:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:39:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:39:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:39:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:39:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:39:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:39:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:39:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:39:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:39:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:39:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:39:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:39:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:39:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:39:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:39:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:39:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:39:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:39:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:39:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:39:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:39:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.status 2023-12-06T15:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.status 2023-12-06T15:39:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:39:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:39:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.status 2023-12-06T15:39:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.status 2023-12-06T15:39:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:39:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:39:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-06T15:39:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:39:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:39:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:39:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-06T15:40:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 29 2023-12-06T15:40:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:40:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:40:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:40:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-06T15:40:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:40:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:40:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=29 2023-12-06T15:40:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:40:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:40:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:40:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:40:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:40:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:40:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:40:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:40:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:40:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:40:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:40:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 29 = '' 2023-12-06T15:40:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:40:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:40:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:40:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:40:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:40:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:40:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:40:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:40:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:40:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:40:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 29 = '' 2023-12-06T15:40:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:40:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:40:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:40:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:40:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:40:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:40:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:40:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:40:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:40:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:40:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 29 = '' 2023-12-06T15:40:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:40:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:40:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:40:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:40:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:40:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=29 2023-12-06T15:40:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 29 = 29 2023-12-06T15:40:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:40:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-12-06T15:40:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap7 2023-12-06T15:40:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:40:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:40:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:40:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-12-06T15:40:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2023-12-06T15:40:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T15:40:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:40:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 14 2023-12-06T15:40:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2376d3f3f 2023-12-06T15:40:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2376d3f3f 2023-12-06T15:40:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:40:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:40:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:40:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:40:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:40:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:39:17 2023 2023-12-06T15:40:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:40:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:40:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:40:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:40:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:40:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:40:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:40:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snap6-clean ']' 2023-12-06T15:40:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-06T15:40:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap6-clean 2023-12-06T15:40:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:40:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:40:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:40:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:40:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:40:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:40:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap7 2023-12-06T15:40:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap7 2023-12-06T15:40:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:40:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:40:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:40:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:40:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:40:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:40:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clean_snap_name=snap7-clean 2023-12-06T15:40:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-12-06T15:40:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:40:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap7-clean 2023-12-06T15:40:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-12-06T15:40:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:40:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror test 2023-12-06T15:40:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:40:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:40:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=9 2023-12-06T15:40:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:40:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-06T15:40:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 4604928 bytes 2023-12-06T15:40:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 36411392 bytes 2023-12-06T15:40:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 112822272 bytes 2023-12-06T15:40:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 182294528 bytes 2023-12-06T15:40:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 254906368 bytes 2023-12-06T15:40:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 273058816 bytes 2023-12-06T15:40:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 303859712 bytes 2023-12-06T15:40:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 324423680 bytes 2023-12-06T15:40:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 389343232 bytes 2023-12-06T15:40:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 438220800 bytes 2023-12-06T15:40:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 459358208 bytes 2023-12-06T15:40:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 504597504 bytes 2023-12-06T15:40:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 505208832 bytes 2023-12-06T15:40:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 553107456 bytes 2023-12-06T15:40:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 583923712 bytes 2023-12-06T15:40:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 611795968 bytes 2023-12-06T15:40:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 681768960 bytes 2023-12-06T15:40:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 708882432 bytes 2023-12-06T15:40:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 740433920 bytes 2023-12-06T15:40:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 762828800 bytes 2023-12-06T15:40:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 831531008 bytes 2023-12-06T15:40:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 882211840 bytes 2023-12-06T15:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 928727040 bytes 2023-12-06T15:40:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 969928704 bytes 2023-12-06T15:40:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1068407808 bytes 2023-12-06T15:40:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1078466560 bytes 2023-12-06T15:40:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1156452352 bytes 2023-12-06T15:40:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1215023104 bytes 2023-12-06T15:40:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1272009728 bytes 2023-12-06T15:40:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1272628224 bytes 2023-12-06T15:40:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1359313920 bytes 2023-12-06T15:40:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1384624128 bytes 2023-12-06T15:40:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1409126400 bytes 2023-12-06T15:40:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1419305984 bytes 2023-12-06T15:40:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1445684224 bytes 2023-12-06T15:40:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1475938304 bytes 2023-12-06T15:40:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1517884416 bytes 2023-12-06T15:40:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1548746752 bytes 2023-12-06T15:40:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1593221120 bytes 2023-12-06T15:40:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1694217216 bytes 2023-12-06T15:40:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1748835328 bytes 2023-12-06T15:40:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1814165504 bytes 2023-12-06T15:40:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1844468736 bytes 2023-12-06T15:40:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1845093376 bytes 2023-12-06T15:40:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1858575360 bytes 2023-12-06T15:40:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1924005888 bytes 2023-12-06T15:40:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1947039744 bytes 2023-12-06T15:40:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1995435008 bytes 2023-12-06T15:40:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2017072128 bytes 2023-12-06T15:40:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2064539648 bytes 2023-12-06T15:40:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2088590336 bytes 2023-12-06T15:40:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2106365952 bytes 2023-12-06T15:40:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2181693440 bytes 2023-12-06T15:40:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:40:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:40:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:40:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:40:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-06T15:40:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:40:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:40:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:40:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:40:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:40:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:40:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:40:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:40:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:40:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:40:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:40:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:40:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:40:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:40:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:40:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:40:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:40:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:40:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2023-12-06T15:40:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533588992.0,"last_snapshot_bytes":532599808,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701877149,"remote_snapshot_timestamp":1701877149,"replay_state":"idle"} 2023-12-06T15:40:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2023-12-06T15:40:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:39:58 2023-12-06T15:40:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:40:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:40:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:40:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:40:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:40:04 2023-12-06T15:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:40:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2023-12-06T15:40:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533588992.0,"last_snapshot_bytes":532599808,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701877149,"remote_snapshot_timestamp":1701877149,"replay_state":"idle"} 2023-12-06T15:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2023-12-06T15:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:40:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2023-12-06T15:40:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:40:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'last_update=2023-12-06 15:39:58' 2023-12-06T15:40:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-06 15:39:58' 2023-12-06T15:40:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:40:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2023-12-06 15:39:58' 2023-12-06T15:40:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-06T15:40:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror.mirror_status 2023-12-06T15:40:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:40:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror.mirror_status 2023-12-06T15:40:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-06T15:40:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:40:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2023-12-06T15:40:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-06T15:40:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2023-12-06 15:39:58' 2023-12-06T15:40:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-06T15:40:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2023-12-06T15:40:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:40:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2023-12-06T15:40:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:40:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:40:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:40:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap8 2023-12-06T15:40:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap8 2023-12-06T15:40:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:40:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap8 2023-12-06T15:40:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:40:36.208 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-06T15:40:36.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-06T15:40:36.209 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-06T15:40:36.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:40:36.209+0000 7fc246a8f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 125444) UID: 1000 2023-12-06T15:40:36.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:40:39.210 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-06T15:40:39.210 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:40:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:40:39.210 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-06T15:40:39.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-06T15:40:39.211 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-06T15:40:39.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-06T15:40:39.215 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2023-12-06T15:40:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-06T15:40:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-06T15:40:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:40:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:40:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:40:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-06T15:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-06T15:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:40:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:40:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:40:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:40:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:40:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:40:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:40:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:40:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:40:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:40:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:40:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:40:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:40:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:40:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:40:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:40:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:40:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:40:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:40:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:40:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:40:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:40:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:40:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:40:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:40:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:40:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:40:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:40:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:40:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:40:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:40:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:40:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:40:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:40:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:40:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:40:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:40:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-06T15:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:40:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:40:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-06T15:40:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:40:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:40:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.status 2023-12-06T15:40:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.status 2023-12-06T15:40:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:40:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:40:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.status 2023-12-06T15:40:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.status 2023-12-06T15:40:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:40:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-06T15:40:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:40:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-06T15:40:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 33 2023-12-06T15:40:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:40:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:40:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:40:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-06T15:40:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:40:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:40:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=33 2023-12-06T15:40:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:40:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:40:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:40:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:40:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:40:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:40:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:40:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:40:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:40:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:40:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:40:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 33 = '' 2023-12-06T15:40:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:40:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:40:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:40:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:40:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:40:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:40:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:40:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:40:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:40:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:40:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 33 = '' 2023-12-06T15:40:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:40:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:40:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:40:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:40:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:40:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:40:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:40:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:40:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:40:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:40:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 33 = '' 2023-12-06T15:40:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:40:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:40:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:40:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:40:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:40:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:40:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:40:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:40:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:40:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:40:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 33 = '' 2023-12-06T15:40:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:40:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2023-12-06T15:40:54.306 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.822323213269926) > weight (0.25) 2023-12-06T15:40:54.306 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7758675059071554) > weight (0.25) 2023-12-06T15:40:54.306 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4275124292450695) > weight (0.25) 2023-12-06T15:40:54.306 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-06T15:40:54.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-06T15:40:54.307 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-06T15:40:54.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr:2023-12-06T15:40:54.306+0000 7fae78263700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 119483) UID: 1000 2023-12-06T15:40:54.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:40:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:40:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:40:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:40:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:40:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=33 2023-12-06T15:40:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 33 = 33 2023-12-06T15:40:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:40:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-12-06T15:40:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:40:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap8 2023-12-06T15:40:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:40:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:40:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:40:56.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-12-06T15:40:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2023-12-06T15:40:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T15:40:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:40:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 16 2023-12-06T15:40:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2376d3f3f 2023-12-06T15:40:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2376d3f3f 2023-12-06T15:40:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:40:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:40:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:40:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:40:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:40:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:39:17 2023 2023-12-06T15:40:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:40:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:40:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:40:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:40:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:40:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:40:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:40:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snap7-clean ']' 2023-12-06T15:40:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-06T15:40:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap7-clean 2023-12-06T15:40:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:40:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:40:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:40:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:40:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:40:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:40:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap8 2023-12-06T15:40:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:40:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:40:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap8 2023-12-06T15:40:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:40:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:40:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:40:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:40:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:40:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:41:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:00.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:41:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clean_snap_name=snap8-clean 2023-12-06T15:41:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-12-06T15:41:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:41:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap8-clean 2023-12-06T15:41:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-12-06T15:41:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:41:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror test 2023-12-06T15:41:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:41:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:41:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=6 2023-12-06T15:41:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:41:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-06T15:41:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 89504768 bytes 2023-12-06T15:41:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 114834432 bytes 2023-12-06T15:41:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 153485312 bytes 2023-12-06T15:41:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 173999104 bytes 2023-12-06T15:41:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 215666688 bytes 2023-12-06T15:41:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 284892160 bytes 2023-12-06T15:41:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 305046528 bytes 2023-12-06T15:41:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 410595328 bytes 2023-12-06T15:41:03.308 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-06T15:41:03.308 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:41:03.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:41:03.308 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-06T15:41:03.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-06T15:41:03.309 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-06T15:41:03.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-06T15:41:03.312 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2023-12-06T15:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 436168704 bytes 2023-12-06T15:41:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 498221056 bytes 2023-12-06T15:41:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 524500992 bytes 2023-12-06T15:41:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 570976256 bytes 2023-12-06T15:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 624714752 bytes 2023-12-06T15:41:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 625333248 bytes 2023-12-06T15:41:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 668305408 bytes 2023-12-06T15:41:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 690644992 bytes 2023-12-06T15:41:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 727859200 bytes 2023-12-06T15:41:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 764938240 bytes 2023-12-06T15:41:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 855477248 bytes 2023-12-06T15:41:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 881282048 bytes 2023-12-06T15:41:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 899045376 bytes 2023-12-06T15:41:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 975583232 bytes 2023-12-06T15:41:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1018371072 bytes 2023-12-06T15:41:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1075645440 bytes 2023-12-06T15:41:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1136377856 bytes 2023-12-06T15:41:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1140736000 bytes 2023-12-06T15:41:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1228411904 bytes 2023-12-06T15:41:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1285995520 bytes 2023-12-06T15:41:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1303883776 bytes 2023-12-06T15:41:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1304507392 bytes 2023-12-06T15:41:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1410679808 bytes 2023-12-06T15:41:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1465068544 bytes 2023-12-06T15:41:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1479886848 bytes 2023-12-06T15:41:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1486124032 bytes 2023-12-06T15:41:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:41:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:41:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:41:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:41:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-06T15:41:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:41:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:41:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:41:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:41:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:41:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:41:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:41:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:41:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:41:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:41:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:41:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:41:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:41:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:41:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:41:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:41:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:41:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:41:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2023-12-06T15:41:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":17659528.53,"bytes_per_snapshot":526364160.0,"last_snapshot_bytes":529785856,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701877249,"remote_snapshot_timestamp":1701877249,"replay_state":"idle"} 2023-12-06T15:41:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2023-12-06T15:41:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:40:58 2023-12-06T15:41:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:41:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:41:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:41:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:41:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:41:04 2023-12-06T15:41:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:41:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2023-12-06T15:41:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:41:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":17659528.53,"bytes_per_snapshot":526364160.0,"last_snapshot_bytes":529785856,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701877249,"remote_snapshot_timestamp":1701877249,"replay_state":"idle"} 2023-12-06T15:41:08.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:41:08.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:41:08.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2023-12-06T15:41:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:41:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:41:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2023-12-06T15:41:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:41:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'last_update=2023-12-06 15:40:58' 2023-12-06T15:41:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-06 15:40:58' 2023-12-06T15:41:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:41:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2023-12-06 15:40:58' 2023-12-06T15:41:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-06T15:41:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror.mirror_status 2023-12-06T15:41:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:41:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror.mirror_status 2023-12-06T15:41:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-06T15:41:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:41:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2023-12-06T15:41:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-06T15:41:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2023-12-06 15:40:58' 2023-12-06T15:41:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-06T15:41:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2023-12-06T15:41:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:41:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2023-12-06T15:41:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:41:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:41:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:41:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap9 2023-12-06T15:41:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap9 2023-12-06T15:41:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:41:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap9 2023-12-06T15:41:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:41:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-06T15:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-06T15:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:41:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:41:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-06T15:41:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:41:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:41:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:41:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-06T15:41:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:41:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:41:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:41:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:41:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:41:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:41:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:41:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:41:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:41:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:41:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:41:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:41:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:41:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:41:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:41:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:41:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:41:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:41:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:41:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:41:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:41:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:41:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:41:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:41:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:41:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:41:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:41:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:41:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:41:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:41:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:41:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:41:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:41:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:41:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:41:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:41:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:41:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:41:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:41:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:41:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:41:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:41:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:41:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:41:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:41:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:41:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:41:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:41:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:41:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:41:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:41:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:41:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-06T15:41:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:41:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:41:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:41:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:41:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-06T15:41:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:41:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:41:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.status 2023-12-06T15:41:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.status 2023-12-06T15:41:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:41:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:41:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.status 2023-12-06T15:41:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.status 2023-12-06T15:41:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:41:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:41:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-06T15:41:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:41:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-06T15:41:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 37 2023-12-06T15:41:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:41:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:41:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:41:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-06T15:41:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:41:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:41:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=37 2023-12-06T15:41:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:41:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:41:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:41:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:41:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:41:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:41:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:41:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:41:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:41:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:41:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:41:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:41:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:41:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 37 = '' 2023-12-06T15:41:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:41:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:41:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:41:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:41:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:41:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:41:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:41:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:41:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:41:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 37 = '' 2023-12-06T15:41:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:41:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:41:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:41:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:41:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:41:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:41:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:41:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:41:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:41:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:41:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:41:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:41:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 37 = '' 2023-12-06T15:41:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:41:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:41:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:41:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:41:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:41:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:41:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:41:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=37 2023-12-06T15:41:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 37 = 37 2023-12-06T15:41:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:41:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-12-06T15:41:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap9 2023-12-06T15:41:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:41:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:41:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:41:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-12-06T15:41:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2023-12-06T15:41:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 18 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2376d3f3f 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2376d3f3f 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:40:57 2023 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:41:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:41:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snap8-clean ']' 2023-12-06T15:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-06T15:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap8-clean 2023-12-06T15:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:41:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:41:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:41:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:41:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap9 2023-12-06T15:41:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap9 2023-12-06T15:41:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:41:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:41:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:41:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:41:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:41:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:41:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:41:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clean_snap_name=snap9-clean 2023-12-06T15:41:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-12-06T15:41:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:41:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap9-clean 2023-12-06T15:41:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-12-06T15:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror test 2023-12-06T15:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:41:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=9 2023-12-06T15:41:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:41:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-06T15:41:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 71895040 bytes 2023-12-06T15:41:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 111594496 bytes 2023-12-06T15:41:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 129788928 bytes 2023-12-06T15:41:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 158101504 bytes 2023-12-06T15:41:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 159424512 bytes 2023-12-06T15:41:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 173650944 bytes 2023-12-06T15:41:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 213438464 bytes 2023-12-06T15:41:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 214407168 bytes 2023-12-06T15:41:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 243997696 bytes 2023-12-06T15:41:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 251927552 bytes 2023-12-06T15:41:51.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 326183936 bytes 2023-12-06T15:41:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 378585088 bytes 2023-12-06T15:41:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 424417280 bytes 2023-12-06T15:41:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 440240128 bytes 2023-12-06T15:41:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 512997376 bytes 2023-12-06T15:41:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 535061504 bytes 2023-12-06T15:41:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 577606656 bytes 2023-12-06T15:41:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 627666944 bytes 2023-12-06T15:41:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 644643840 bytes 2023-12-06T15:41:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 675586048 bytes 2023-12-06T15:41:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 721516544 bytes 2023-12-06T15:41:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 755731456 bytes 2023-12-06T15:41:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 787867648 bytes 2023-12-06T15:41:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 821364736 bytes 2023-12-06T15:41:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 848353280 bytes 2023-12-06T15:41:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 874018816 bytes 2023-12-06T15:41:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 901535744 bytes 2023-12-06T15:41:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 916939776 bytes 2023-12-06T15:41:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 934509568 bytes 2023-12-06T15:41:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1034967040 bytes 2023-12-06T15:41:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1053301760 bytes 2023-12-06T15:41:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1054966784 bytes 2023-12-06T15:41:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1071495168 bytes 2023-12-06T15:41:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1109685248 bytes 2023-12-06T15:41:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1129980928 bytes 2023-12-06T15:41:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1157409792 bytes 2023-12-06T15:41:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1173895168 bytes 2023-12-06T15:41:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1218914304 bytes 2023-12-06T15:41:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1248542720 bytes 2023-12-06T15:41:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1285608448 bytes 2023-12-06T15:41:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1340659712 bytes 2023-12-06T15:41:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1403370496 bytes 2023-12-06T15:41:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1458013184 bytes 2023-12-06T15:41:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1501405184 bytes 2023-12-06T15:41:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1531425792 bytes 2023-12-06T15:41:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1624296448 bytes 2023-12-06T15:41:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1692187648 bytes 2023-12-06T15:41:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1710280704 bytes 2023-12-06T15:41:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1743633408 bytes 2023-12-06T15:41:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1798405120 bytes 2023-12-06T15:41:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 1863819264 bytes 2023-12-06T15:41:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 1882202112 bytes 2023-12-06T15:41:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 1934104576 bytes 2023-12-06T15:41:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 1987103744 bytes 2023-12-06T15:41:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2010211328 bytes 2023-12-06T15:41:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2014051328 bytes 2023-12-06T15:41:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:41:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:41:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:41:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:41:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-06T15:41:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:41:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:41:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:41:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:41:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:41:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-06T15:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2023-12-06T15:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2023-12-06T15:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-06T15:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-06T15:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2023-12-06T15:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-06T15:41:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:41:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:test: 2023-12-06T15:41:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:41:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2023-12-06T15:41:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":17206681.6,"bytes_per_snapshot":522993152.0,"last_snapshot_bytes":516200448,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701877300,"remote_snapshot_timestamp":1701877300,"replay_state":"idle"} 2023-12-06T15:41:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2023-12-06T15:41:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:41:58 2023-12-06T15:41:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2023-12-06T15:41:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2023-12-06T15:41:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2023-12-06T15:41:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2023-12-06T15:41:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2023-12-06 15:41:34 2023-12-06T15:41:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:41:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2023-12-06T15:41:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:41:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":17206681.6,"bytes_per_snapshot":522993152.0,"last_snapshot_bytes":516200448,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701877300,"remote_snapshot_timestamp":1701877300,"replay_state":"idle"} 2023-12-06T15:41:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:41:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:41:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2023-12-06T15:41:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:41:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:41:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2023-12-06T15:41:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.mirror_status 2023-12-06T15:41:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'last_update=2023-12-06 15:41:58' 2023-12-06T15:41:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-06 15:41:58' 2023-12-06T15:41:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:41:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2023-12-06T15:41:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2023-12-06 15:41:58' 2023-12-06T15:41:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-06T15:41:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror.mirror_status 2023-12-06T15:41:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:41:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror.mirror_status 2023-12-06T15:41:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-06T15:41:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:41:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2023-12-06T15:41:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-06T15:41:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2023-12-06 15:41:58' 2023-12-06T15:41:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-06T15:41:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2023-12-06T15:41:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2023-12-06T15:41:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2023-12-06T15:41:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2023-12-06T15:41:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:41:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:41:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap10 2023-12-06T15:41:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap10 2023-12-06T15:41:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:41:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:41:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:41:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap10 2023-12-06T15:41:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:42:01.215 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-06T15:42:01.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-06T15:42:01.216 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-06T15:42:01.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:42:01.215+0000 7f8bb826d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 126610) UID: 1000 2023-12-06T15:42:01.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:42:05.216 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-06T15:42:05.217 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:42:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:42:05.217 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-06T15:42:05.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-06T15:42:05.218 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-06T15:42:05.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-06T15:42:05.221 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2023-12-06T15:42:28.313 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8777535500097987) > weight (0.25) 2023-12-06T15:42:28.314 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9121003225577384) > weight (0.25) 2023-12-06T15:42:28.314 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.41477146069404236) > weight (0.25) 2023-12-06T15:42:28.314 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6783217301126038) > weight (0.25) 2023-12-06T15:42:28.314 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2023-12-06T15:42:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-06T15:42:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-06T15:42:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:42:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:42:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:42:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:42:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-06T15:42:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:42:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:42:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:42:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-06T15:42:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:42:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:42:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:42:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:42:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:42:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:42:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:42:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:42:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:42:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:42:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:42:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:42:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:42:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:42:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:42:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:42:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:42:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:42:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:42:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:42:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:42:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:42:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:42:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:42:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:42:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:42:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:42:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:42:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:42:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:42:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:42:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:42:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:42:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:42:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:42:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:42:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:42:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:42:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:42:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:42:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:42:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:42:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:42:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:42:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:42:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:42:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:42:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:42:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:42:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:42:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:42:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:42:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:42:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:42:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:42:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:42:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:42:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:42:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:42:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:42:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:42:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:42:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:42:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:42:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:42:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:42:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:42:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:42:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:42:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:42:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:42:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:42:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:42:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:42:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:42:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:42:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:42:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:42:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:42:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.status 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.status 2023-12-06T15:42:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:42:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:42:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.status 2023-12-06T15:42:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.status 2023-12-06T15:42:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:42:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-06T15:42:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:42:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-06T15:42:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 41 2023-12-06T15:42:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:42:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:42:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:42:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-06T15:42:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:42:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:42:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=41 2023-12-06T15:42:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:42:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:42:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:42:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:42:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:42:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:42:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:42:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:42:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:42:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:42:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 41 = '' 2023-12-06T15:42:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:42:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:42:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:42:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:42:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:42:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:42:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:42:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:42:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:42:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:42:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 41 = '' 2023-12-06T15:42:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:42:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:42:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:42:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:42:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:42:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:42:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:42:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:42:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:42:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:42:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 41 = '' 2023-12-06T15:42:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:42:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:42:34.749 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2023-12-06T15:42:34.748+0000 7f6458ca5700 -1 reset not still connected to 0x5594efcc7450 2023-12-06T15:42:34.749 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2023-12-06T15:42:34.748+0000 7f6458ca5700 -1 reset not still connected to 0x5594f4521ad0 2023-12-06T15:42:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:42:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:42:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:42:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=41 2023-12-06T15:42:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 41 = 41 2023-12-06T15:42:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-12-06T15:42:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap10 2023-12-06T15:42:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:42:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:42:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:42:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-12-06T15:42:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2023-12-06T15:42:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T15:42:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:42:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 20 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2376d3f3f 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2376d3f3f 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:40:57 2023 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:42:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:42:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snap9-clean ']' 2023-12-06T15:42:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-06T15:42:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap9-clean 2023-12-06T15:42:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:42:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap10 2023-12-06T15:42:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap10 2023-12-06T15:42:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:42:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clean_snap_name=snap10-clean 2023-12-06T15:42:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-12-06T15:42:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:42:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap10-clean 2023-12-06T15:42:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-12-06T15:42:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-06T15:42:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-06T15:42:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:42:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:42:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:42:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:42:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-06T15:42:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:42:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:42:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:42:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-06T15:42:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:42:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:42:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:42:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:42:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:42:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:42:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:42:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:42:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:42:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:42:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:42:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:42:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:42:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:42:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:42:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:42:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:42:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:42:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:42:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:42:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:42:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:42:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:42:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:42:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:42:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:42:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:42:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:42:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:42:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:42:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-06T15:42:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-06T15:42:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:42:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:42:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:42:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-06T15:42:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:42:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:42:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:42:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:42:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:42:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:42:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:42:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:42:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:42:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:42:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:42:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:42:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:42:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:42:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:42:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:42:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:42:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:42:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-06T15:42:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:42:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:42:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:42:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:42:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:42:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.status 2023-12-06T15:42:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.status 2023-12-06T15:42:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:42:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:42:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.status 2023-12-06T15:42:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.status 2023-12-06T15:42:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:42:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-06T15:42:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:42:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-06T15:42:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 44 2023-12-06T15:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.status 2023-12-06T15:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-06T15:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:42:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:42:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=44 2023-12-06T15:42:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:42:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:42:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:42:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:42:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:42:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:42:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:42:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:42:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:42:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:42:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:42:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 44 = '' 2023-12-06T15:42:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:42:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:42:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:42:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:42:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:42:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:42:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:42:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:42:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:42:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:42:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 44 = '' 2023-12-06T15:42:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:42:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:42:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.status 2023-12-06T15:42:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-06T15:42:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:42:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:42:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=44 2023-12-06T15:42:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 44 = 44 2023-12-06T15:42:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-12-06T15:42:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:42:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap10-clean 2023-12-06T15:42:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:42:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:42:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:42:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 21 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2376d3f3f 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2376d3f3f 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:42:37 2023 2023-12-06T15:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:34:00 2023 2023-12-06T15:42:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:42:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:42:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:42:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f83e8239-9713-4cca-a2da-e11c5afe1cb3 2023-12-06T15:42:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:42:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 10 2023-12-06T15:42:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:42:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap1 2023-12-06T15:42:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap1 2023-12-06T15:42:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap1 2023-12-06T15:42:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:42:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap1-clean 2023-12-06T15:42:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-06T15:42:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap1-clean 2023-12-06T15:42:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:42:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:42:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap2 2023-12-06T15:42:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap2 2023-12-06T15:42:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2023-12-06T15:42:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:42:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap2-clean 2023-12-06T15:42:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-06T15:42:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2-clean 2023-12-06T15:42:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:42:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:42:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap3 2023-12-06T15:42:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap3 2023-12-06T15:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap3 2023-12-06T15:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap3-clean 2023-12-06T15:42:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-06T15:42:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap3-clean 2023-12-06T15:42:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:42:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:42:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap4 2023-12-06T15:42:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap4 2023-12-06T15:42:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap4 2023-12-06T15:42:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:42:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:42:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap4-clean 2023-12-06T15:42:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-06T15:42:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:42:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:42:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap4-clean 2023-12-06T15:42:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:42:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:42:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:42:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:42:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap5 2023-12-06T15:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap5 2023-12-06T15:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap5 2023-12-06T15:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap5-clean 2023-12-06T15:43:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-06T15:43:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap5-clean 2023-12-06T15:43:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:43:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap6 2023-12-06T15:43:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap6 2023-12-06T15:43:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap6 2023-12-06T15:43:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:43:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap6-clean 2023-12-06T15:43:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-06T15:43:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap6-clean 2023-12-06T15:43:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:43:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap7 2023-12-06T15:43:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap7 2023-12-06T15:43:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap7 2023-12-06T15:43:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:43:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap7-clean 2023-12-06T15:43:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-06T15:43:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap7-clean 2023-12-06T15:43:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:43:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap8 2023-12-06T15:43:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap8 2023-12-06T15:43:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap8 2023-12-06T15:43:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:43:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap8-clean 2023-12-06T15:43:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-06T15:43:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap8-clean 2023-12-06T15:43:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:43:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap9 2023-12-06T15:43:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap9 2023-12-06T15:43:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap9 2023-12-06T15:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap9-clean 2023-12-06T15:43:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-06T15:43:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap9-clean 2023-12-06T15:43:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:43:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap10 2023-12-06T15:43:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap10 2023-12-06T15:43:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap10 2023-12-06T15:43:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:43:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap10-clean 2023-12-06T15:43:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror test snap10-clean 2023-12-06T15:43:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap10-clean 2023-12-06T15:43:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:43:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:20.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:20.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:20.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export 2023-12-06T15:43:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-06T15:43:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-test.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-test.export 2023-12-06T15:43:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 10 2023-12-06T15:43:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap1 2023-12-06T15:43:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-12-06T15:43:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2023-12-06T15:43:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-12-06T15:43:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap1-clean 2023-12-06T15:43:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-12-06T15:43:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1-clean 2023-12-06T15:43:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-12-06T15:43:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap2 2023-12-06T15:43:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-12-06T15:43:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2023-12-06T15:43:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-12-06T15:43:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap2-clean 2023-12-06T15:43:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-12-06T15:43:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2-clean 2023-12-06T15:43:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-12-06T15:43:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap3 2023-12-06T15:43:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-12-06T15:43:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap3 2023-12-06T15:43:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-12-06T15:43:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap3-clean 2023-12-06T15:43:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-12-06T15:43:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap3-clean 2023-12-06T15:43:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-12-06T15:43:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap4 2023-12-06T15:43:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-12-06T15:43:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap4 2023-12-06T15:43:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-12-06T15:43:28.315 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9119062611703641) > weight (0.25) 2023-12-06T15:43:28.315 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8121669880858096) > weight (0.25) 2023-12-06T15:43:28.315 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8975857814547019) > weight (0.25) 2023-12-06T15:43:28.315 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.26049112738035707) > weight (0.25) 2023-12-06T15:43:28.315 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-06T15:43:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap4-clean 2023-12-06T15:43:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-12-06T15:43:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap4-clean 2023-12-06T15:43:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-12-06T15:43:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap5 2023-12-06T15:43:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-12-06T15:43:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap5 2023-12-06T15:43:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-12-06T15:43:30.223 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8612485468895502) > weight (0.25) 2023-12-06T15:43:30.223 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.325132941998565) > weight (0.25) 2023-12-06T15:43:30.223 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.34412983423919896) > weight (0.25) 2023-12-06T15:43:30.223 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.2903570889411553) > weight (0.25) 2023-12-06T15:43:30.223 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2023-12-06T15:43:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap5-clean 2023-12-06T15:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-12-06T15:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap5-clean 2023-12-06T15:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-12-06T15:43:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap6 2023-12-06T15:43:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-12-06T15:43:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap6 2023-12-06T15:43:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-12-06T15:43:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap6-clean 2023-12-06T15:43:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-12-06T15:43:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap6-clean 2023-12-06T15:43:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-12-06T15:43:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap7 2023-12-06T15:43:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-12-06T15:43:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap7 2023-12-06T15:43:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-12-06T15:43:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap7-clean 2023-12-06T15:43:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-12-06T15:43:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap7-clean 2023-12-06T15:43:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-12-06T15:43:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap8 2023-12-06T15:43:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-12-06T15:43:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap8 2023-12-06T15:43:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-12-06T15:43:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap8-clean 2023-12-06T15:43:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-12-06T15:43:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap8-clean 2023-12-06T15:43:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-12-06T15:43:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap9 2023-12-06T15:43:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-12-06T15:43:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap9 2023-12-06T15:43:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-12-06T15:43:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap9-clean 2023-12-06T15:43:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-12-06T15:43:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap9-clean 2023-12-06T15:43:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-12-06T15:43:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T15:43:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap10 2023-12-06T15:43:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-12-06T15:43:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap10 2023-12-06T15:43:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-12-06T15:43:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap10-clean 2023-12-06T15:43:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-12-06T15:43:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap10-clean 2023-12-06T15:43:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-12-06T15:43:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:43:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test 2023-12-06T15:43:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T15:43:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T15:43:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test 2023-12-06T15:43:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-12-06T15:43:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-12-06T15:43:41.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:43:41.873+0000 7fd1f83f2700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a06c8a000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T15:43:41.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:43:41.873+0000 7fd1febff700 -1 rbd::mirror::ImageReplayer: 0x558a02b2e280 [3/f83e8239-9713-4cca-a2da-e11c5afe1cb3] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T15:43:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-06T15:43:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-06T15:43:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T15:43:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-06T15:43:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T15:43:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T15:43:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted '' 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2023-12-06T15:43:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-06T15:43:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T15:43:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-06T15:43:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T15:43:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T15:43:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T15:43:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: create many images' 2023-12-06T15:43:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.Q1BJHHgE3s/rbd-mirror.test.log 2023-12-06T15:43:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2023-12-06T15:43:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2023-12-06 15:43:44 TEST: create many images 2023-12-06T15:43:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2023-12-06 15:43:44 TEST: create many images 2023-12-06T15:43:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap 2023-12-06T15:43:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 50 2023-12-06T15:43:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:43:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_1 2023-12-06T15:43:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-12-06T15:43:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:43:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_1 128M 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-12-06T15:43:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:43:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-12-06T15:43:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:43:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:43:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-06T15:43:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:43:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-06T15:43:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_1': 2023-12-06T15:43:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:43:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:43:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:43:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 124929a23bc5 2023-12-06T15:43:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.124929a23bc5 2023-12-06T15:43:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:43:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:43:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:43:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:43:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:43:44 2023 2023-12-06T15:43:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:43:44 2023 2023-12-06T15:43:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:43:44 2023 2023-12-06T15:43:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:43:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:43:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 0b9a1c1c-7b41-4db7-9c1f-9f9fa3190aa6 2023-12-06T15:43:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:43:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:43:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-06T15:43:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:43:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:43:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=11 2023-12-06T15:43:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:43:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 45957120 bytes 2023-12-06T15:43:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 75646976 bytes 2023-12-06T15:43:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 87885824 bytes 2023-12-06T15:43:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 112608256 bytes 2023-12-06T15:43:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 144303104 bytes 2023-12-06T15:43:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 241216512 bytes 2023-12-06T15:43:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 294351872 bytes 2023-12-06T15:43:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 329547776 bytes 2023-12-06T15:43:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 388277248 bytes 2023-12-06T15:43:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 427158528 bytes 2023-12-06T15:43:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 466810880 bytes 2023-12-06T15:43:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 482996224 bytes 2023-12-06T15:43:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 484840448 bytes 2023-12-06T15:43:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 526424064 bytes 2023-12-06T15:43:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 590181376 bytes 2023-12-06T15:43:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 645444608 bytes 2023-12-06T15:43:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 732350464 bytes 2023-12-06T15:43:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 778427392 bytes 2023-12-06T15:43:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 797279232 bytes 2023-12-06T15:43:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 844263424 bytes 2023-12-06T15:43:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 852417536 bytes 2023-12-06T15:43:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 911243264 bytes 2023-12-06T15:43:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 927034368 bytes 2023-12-06T15:43:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 984124416 bytes 2023-12-06T15:43:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1006276608 bytes 2023-12-06T15:43:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1031414784 bytes 2023-12-06T15:43:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1095079936 bytes 2023-12-06T15:43:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1159031808 bytes 2023-12-06T15:43:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1248722944 bytes 2023-12-06T15:43:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1322867712 bytes 2023-12-06T15:43:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1339034624 bytes 2023-12-06T15:43:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1379309568 bytes 2023-12-06T15:43:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1379913728 bytes 2023-12-06T15:43:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1410384896 bytes 2023-12-06T15:43:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1453270016 bytes 2023-12-06T15:43:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1501807616 bytes 2023-12-06T15:43:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1539592192 bytes 2023-12-06T15:43:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1611274240 bytes 2023-12-06T15:43:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1611892736 bytes 2023-12-06T15:43:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1630997504 bytes 2023-12-06T15:43:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1672938496 bytes 2023-12-06T15:43:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1734193152 bytes 2023-12-06T15:43:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1807349760 bytes 2023-12-06T15:43:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1846790144 bytes 2023-12-06T15:43:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1863286784 bytes 2023-12-06T15:43:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1871903744 bytes 2023-12-06T15:43:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1899495424 bytes 2023-12-06T15:43:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1914521600 bytes 2023-12-06T15:43:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1993358336 bytes 2023-12-06T15:43:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2045266944 bytes 2023-12-06T15:43:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2052533248 bytes 2023-12-06T15:43:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2053153792 bytes 2023-12-06T15:43:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2117026816 bytes 2023-12-06T15:43:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2146058240 bytes 2023-12-06T15:43:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2184879104 bytes 2023-12-06T15:43:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2185477120 bytes 2023-12-06T15:43:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2200469504 bytes 2023-12-06T15:43:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2253319168 bytes 2023-12-06T15:43:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2271246336 bytes 2023-12-06T15:43:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2325642240 bytes 2023-12-06T15:43:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2360120320 bytes 2023-12-06T15:43:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2399672320 bytes 2023-12-06T15:43:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2456627200 bytes 2023-12-06T15:43:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2465527808 bytes 2023-12-06T15:43:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 2550639616 bytes 2023-12-06T15:43:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:43:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:43:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:43:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:43:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:43:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_2 2023-12-06T15:43:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-12-06T15:43:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:43:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_2 128M 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:43:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:43:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:43:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-12-06T15:43:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:43:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-12-06T15:43:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:43:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:43:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-06T15:43:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:43:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-06T15:43:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_2': 2023-12-06T15:43:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 125392abc45e 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.125392abc45e 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:43:55 2023 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:43:55 2023 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:43:55 2023 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:43:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: ddc26196-98fe-465d-a33a-3c9e85520f7d 2023-12-06T15:43:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:43:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:43:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-06T15:43:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:43:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:43:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:43:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:43:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=9 2023-12-06T15:43:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:43:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:43:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 68310016 bytes 2023-12-06T15:43:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 118917120 bytes 2023-12-06T15:43:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 125641728 bytes 2023-12-06T15:43:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 149709824 bytes 2023-12-06T15:43:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 184283136 bytes 2023-12-06T15:43:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 207399936 bytes 2023-12-06T15:43:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 262264832 bytes 2023-12-06T15:43:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 334653440 bytes 2023-12-06T15:43:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 375332864 bytes 2023-12-06T15:43:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 436419584 bytes 2023-12-06T15:43:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 461714432 bytes 2023-12-06T15:43:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 506701824 bytes 2023-12-06T15:43:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 533850112 bytes 2023-12-06T15:43:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 554148864 bytes 2023-12-06T15:43:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 554750976 bytes 2023-12-06T15:43:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 569818112 bytes 2023-12-06T15:43:59.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 603633664 bytes 2023-12-06T15:43:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 636581888 bytes 2023-12-06T15:43:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 665658368 bytes 2023-12-06T15:43:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 761634816 bytes 2023-12-06T15:44:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 818798592 bytes 2023-12-06T15:44:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 845421568 bytes 2023-12-06T15:44:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 909524992 bytes 2023-12-06T15:44:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 997710848 bytes 2023-12-06T15:44:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1034085376 bytes 2023-12-06T15:44:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1085716480 bytes 2023-12-06T15:44:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1131676672 bytes 2023-12-06T15:44:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1205852160 bytes 2023-12-06T15:44:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1229989888 bytes 2023-12-06T15:44:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1274356736 bytes 2023-12-06T15:44:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1307685888 bytes 2023-12-06T15:44:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1321987072 bytes 2023-12-06T15:44:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1330707456 bytes 2023-12-06T15:44:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1358036992 bytes 2023-12-06T15:44:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1444740096 bytes 2023-12-06T15:44:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1445330944 bytes 2023-12-06T15:44:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1495924736 bytes 2023-12-06T15:44:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1538498560 bytes 2023-12-06T15:44:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1579776000 bytes 2023-12-06T15:44:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1706557440 bytes 2023-12-06T15:44:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1730504704 bytes 2023-12-06T15:44:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1797282816 bytes 2023-12-06T15:44:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1809500160 bytes 2023-12-06T15:44:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1864497152 bytes 2023-12-06T15:44:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1918384128 bytes 2023-12-06T15:44:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1952587776 bytes 2023-12-06T15:44:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1984722944 bytes 2023-12-06T15:44:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2012924928 bytes 2023-12-06T15:44:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2031068160 bytes 2023-12-06T15:44:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2080942080 bytes 2023-12-06T15:44:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2160684032 bytes 2023-12-06T15:44:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2213839872 bytes 2023-12-06T15:44:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:44:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:44:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:44:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:44:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_3 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_3 128M 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:44:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:44:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:44:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:44:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-12-06T15:44:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:44:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-12-06T15:44:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:44:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-06T15:44:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:44:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-06T15:44:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_3': 2023-12-06T15:44:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:44:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:44:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:44:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 125c3c7a9d1f 2023-12-06T15:44:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.125c3c7a9d1f 2023-12-06T15:44:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:44:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:44:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:44:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:44:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:05 2023 2023-12-06T15:44:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:05 2023 2023-12-06T15:44:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:05 2023 2023-12-06T15:44:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:44:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:44:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 82a11cbd-ed20-4f4a-8948-aac80e8e0a67 2023-12-06T15:44:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:44:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:44:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-06T15:44:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:44:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:44:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=5 2023-12-06T15:44:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:44:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:44:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 54575104 bytes 2023-12-06T15:44:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 114780160 bytes 2023-12-06T15:44:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 165963776 bytes 2023-12-06T15:44:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 210543616 bytes 2023-12-06T15:44:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 232608768 bytes 2023-12-06T15:44:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 316189696 bytes 2023-12-06T15:44:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 347658240 bytes 2023-12-06T15:44:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 403510272 bytes 2023-12-06T15:44:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 461945856 bytes 2023-12-06T15:44:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 542174208 bytes 2023-12-06T15:44:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 555639808 bytes 2023-12-06T15:44:09.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 605358080 bytes 2023-12-06T15:44:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 651345920 bytes 2023-12-06T15:44:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 681033728 bytes 2023-12-06T15:44:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 753647616 bytes 2023-12-06T15:44:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 823947264 bytes 2023-12-06T15:44:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 868834304 bytes 2023-12-06T15:44:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 939211776 bytes 2023-12-06T15:44:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 939833344 bytes 2023-12-06T15:44:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 1008535552 bytes 2023-12-06T15:44:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 1073065984 bytes 2023-12-06T15:44:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1116959744 bytes 2023-12-06T15:44:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1139656704 bytes 2023-12-06T15:44:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1208221696 bytes 2023-12-06T15:44:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1247878144 bytes 2023-12-06T15:44:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1273955328 bytes 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_4 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_4 128M 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-12-06T15:44:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:44:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-12-06T15:44:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:44:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-06T15:44:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:44:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-06T15:44:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_4': 2023-12-06T15:44:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:44:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:44:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 12662429c66c 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.12662429c66c 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:11 2023 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:11 2023 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:11 2023 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: b7f20ec4-f4b1-4e5b-a31f-db50e7727e1c 2023-12-06T15:44:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:44:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:44:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-06T15:44:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:44:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:44:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=9 2023-12-06T15:44:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:44:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:44:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 47841280 bytes 2023-12-06T15:44:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 76336128 bytes 2023-12-06T15:44:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 99897344 bytes 2023-12-06T15:44:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 162135040 bytes 2023-12-06T15:44:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 204602368 bytes 2023-12-06T15:44:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 205200384 bytes 2023-12-06T15:44:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 221752320 bytes 2023-12-06T15:44:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 244917248 bytes 2023-12-06T15:44:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 315677696 bytes 2023-12-06T15:44:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 317228032 bytes 2023-12-06T15:44:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 334871552 bytes 2023-12-06T15:44:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 345885696 bytes 2023-12-06T15:44:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 388950016 bytes 2023-12-06T15:44:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 389542912 bytes 2023-12-06T15:44:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 440892416 bytes 2023-12-06T15:44:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 495952896 bytes 2023-12-06T15:44:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 512560128 bytes 2023-12-06T15:44:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 533037056 bytes 2023-12-06T15:44:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 583086080 bytes 2023-12-06T15:44:14.801 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:44:14.798+0000 7f44ae2f8700 -1 reset not still connected to 0x559a06c8b380 2023-12-06T15:44:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 613820416 bytes 2023-12-06T15:44:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 675837952 bytes 2023-12-06T15:44:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 724640768 bytes 2023-12-06T15:44:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 765131776 bytes 2023-12-06T15:44:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 815797248 bytes 2023-12-06T15:44:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 886652928 bytes 2023-12-06T15:44:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 953054208 bytes 2023-12-06T15:44:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 976565248 bytes 2023-12-06T15:44:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1026331648 bytes 2023-12-06T15:44:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1047763968 bytes 2023-12-06T15:44:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1158347776 bytes 2023-12-06T15:44:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1200502784 bytes 2023-12-06T15:44:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1239080960 bytes 2023-12-06T15:44:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1334225920 bytes 2023-12-06T15:44:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1366332416 bytes 2023-12-06T15:44:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1434192896 bytes 2023-12-06T15:44:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1477363712 bytes 2023-12-06T15:44:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1480501248 bytes 2023-12-06T15:44:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1509919744 bytes 2023-12-06T15:44:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1573167104 bytes 2023-12-06T15:44:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1606827008 bytes 2023-12-06T15:44:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1620052992 bytes 2023-12-06T15:44:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1671283712 bytes 2023-12-06T15:44:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1703138304 bytes 2023-12-06T15:44:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1762208768 bytes 2023-12-06T15:44:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1779940352 bytes 2023-12-06T15:44:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1791571968 bytes 2023-12-06T15:44:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1848601600 bytes 2023-12-06T15:44:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1897495552 bytes 2023-12-06T15:44:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1953555456 bytes 2023-12-06T15:44:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1976434688 bytes 2023-12-06T15:44:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2013640704 bytes 2023-12-06T15:44:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2059228160 bytes 2023-12-06T15:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2093079552 bytes 2023-12-06T15:44:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2146563072 bytes 2023-12-06T15:44:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:44:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:44:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:44:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:44:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:44:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_5 2023-12-06T15:44:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-12-06T15:44:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_5 128M 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:44:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:44:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:44:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-12-06T15:44:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:44:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-12-06T15:44:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:44:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-06T15:44:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:44:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-06T15:44:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_5': 2023-12-06T15:44:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:44:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:44:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 127089e5a11 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.127089e5a11 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:20 2023 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:20 2023 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:20 2023 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 292f59e4-56d9-46a7-8aff-49cef92c6c9c 2023-12-06T15:44:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:44:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:44:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-06T15:44:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:44:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:44:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=5 2023-12-06T15:44:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:44:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:44:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 70057984 bytes 2023-12-06T15:44:22.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 110971904 bytes 2023-12-06T15:44:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 149199872 bytes 2023-12-06T15:44:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 149806080 bytes 2023-12-06T15:44:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 188993536 bytes 2023-12-06T15:44:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 236819456 bytes 2023-12-06T15:44:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 266423296 bytes 2023-12-06T15:44:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 281207808 bytes 2023-12-06T15:44:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 341720064 bytes 2023-12-06T15:44:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 374719488 bytes 2023-12-06T15:44:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 442439680 bytes 2023-12-06T15:44:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 487683072 bytes 2023-12-06T15:44:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 524967936 bytes 2023-12-06T15:44:24.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 600342528 bytes 2023-12-06T15:44:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 606827520 bytes 2023-12-06T15:44:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 659696640 bytes 2023-12-06T15:44:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 669678592 bytes 2023-12-06T15:44:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 734670848 bytes 2023-12-06T15:44:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 794316800 bytes 2023-12-06T15:44:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 803046400 bytes 2023-12-06T15:44:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 832860160 bytes 2023-12-06T15:44:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 880209920 bytes 2023-12-06T15:44:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 917947392 bytes 2023-12-06T15:44:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 965495808 bytes 2023-12-06T15:44:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 998875136 bytes 2023-12-06T15:44:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1058837504 bytes 2023-12-06T15:44:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1139710976 bytes 2023-12-06T15:44:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:44:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_6 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_6 128M 2023-12-06T15:44:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:44:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:44:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:44:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:44:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-12-06T15:44:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:44:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-12-06T15:44:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:44:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-06T15:44:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:44:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-06T15:44:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_6': 2023-12-06T15:44:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:44:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:44:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:44:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 127cdb65f509 2023-12-06T15:44:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.127cdb65f509 2023-12-06T15:44:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:44:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:44:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:44:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:44:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:27 2023 2023-12-06T15:44:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:27 2023 2023-12-06T15:44:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:27 2023 2023-12-06T15:44:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:44:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:44:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 1b242e7d-b3bc-4380-95c1-3f0e519a2478 2023-12-06T15:44:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:44:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:44:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-06T15:44:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:44:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:44:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=5 2023-12-06T15:44:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:44:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:44:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 54999040 bytes 2023-12-06T15:44:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 141717504 bytes 2023-12-06T15:44:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 155426816 bytes 2023-12-06T15:44:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 184094720 bytes 2023-12-06T15:44:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 268437504 bytes 2023-12-06T15:44:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 344448000 bytes 2023-12-06T15:44:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 351798272 bytes 2023-12-06T15:44:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 352841728 bytes 2023-12-06T15:44:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 388382720 bytes 2023-12-06T15:44:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 389108736 bytes 2023-12-06T15:44:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 452024320 bytes 2023-12-06T15:44:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 480534528 bytes 2023-12-06T15:44:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 504300544 bytes 2023-12-06T15:44:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 576386048 bytes 2023-12-06T15:44:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 641729536 bytes 2023-12-06T15:44:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 681326592 bytes 2023-12-06T15:44:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 714502144 bytes 2023-12-06T15:44:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 733577216 bytes 2023-12-06T15:44:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 792570880 bytes 2023-12-06T15:44:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 868805632 bytes 2023-12-06T15:44:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 931264512 bytes 2023-12-06T15:44:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 932828160 bytes 2023-12-06T15:44:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1016413184 bytes 2023-12-06T15:44:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1090153472 bytes 2023-12-06T15:44:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1184281600 bytes 2023-12-06T15:44:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1184868352 bytes 2023-12-06T15:44:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1185482752 bytes 2023-12-06T15:44:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1209268224 bytes 2023-12-06T15:44:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1233412096 bytes 2023-12-06T15:44:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:44:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_7 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_7 128M 2023-12-06T15:44:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:44:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:44:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:44:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:44:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-12-06T15:44:32.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:44:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-12-06T15:44:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:44:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-06T15:44:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:44:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_7': 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 1286b3f10508 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.1286b3f10508 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:32 2023 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:32 2023 2023-12-06T15:44:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:32 2023 2023-12-06T15:44:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:44:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:44:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: a45295cc-1ac0-4902-a0d3-c086b4cca650 2023-12-06T15:44:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:44:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:44:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-06T15:44:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:44:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:44:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=13 2023-12-06T15:44:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:44:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:44:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 79828992 bytes 2023-12-06T15:44:33.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 114141184 bytes 2023-12-06T15:44:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 145966080 bytes 2023-12-06T15:44:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 202550272 bytes 2023-12-06T15:44:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 253297664 bytes 2023-12-06T15:44:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 266323968 bytes 2023-12-06T15:44:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 275586048 bytes 2023-12-06T15:44:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 307021824 bytes 2023-12-06T15:44:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 394690560 bytes 2023-12-06T15:44:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 445007872 bytes 2023-12-06T15:44:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 479693824 bytes 2023-12-06T15:44:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 498141184 bytes 2023-12-06T15:44:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 592254976 bytes 2023-12-06T15:44:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 645970944 bytes 2023-12-06T15:44:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 705592320 bytes 2023-12-06T15:44:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 712808448 bytes 2023-12-06T15:44:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 761531392 bytes 2023-12-06T15:44:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 779078656 bytes 2023-12-06T15:44:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 884889600 bytes 2023-12-06T15:44:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 930198528 bytes 2023-12-06T15:44:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 968798208 bytes 2023-12-06T15:44:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1023695872 bytes 2023-12-06T15:44:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1085234176 bytes 2023-12-06T15:44:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1110118400 bytes 2023-12-06T15:44:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1189603328 bytes 2023-12-06T15:44:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1226136576 bytes 2023-12-06T15:44:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1266748416 bytes 2023-12-06T15:44:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1309282304 bytes 2023-12-06T15:44:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1350414336 bytes 2023-12-06T15:44:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1389652992 bytes 2023-12-06T15:44:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1410399232 bytes 2023-12-06T15:44:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1490080768 bytes 2023-12-06T15:44:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1511449600 bytes 2023-12-06T15:44:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1580269568 bytes 2023-12-06T15:44:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1650555904 bytes 2023-12-06T15:44:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1742403584 bytes 2023-12-06T15:44:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1784884224 bytes 2023-12-06T15:44:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1816870912 bytes 2023-12-06T15:44:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1837073408 bytes 2023-12-06T15:44:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1868515328 bytes 2023-12-06T15:44:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1921611776 bytes 2023-12-06T15:44:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1971824640 bytes 2023-12-06T15:44:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 2049601536 bytes 2023-12-06T15:44:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 2157812736 bytes 2023-12-06T15:44:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 2179151872 bytes 2023-12-06T15:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 2243149824 bytes 2023-12-06T15:44:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2285740032 bytes 2023-12-06T15:44:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2352270336 bytes 2023-12-06T15:44:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2352887808 bytes 2023-12-06T15:44:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2401942528 bytes 2023-12-06T15:44:42.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2426076160 bytes 2023-12-06T15:44:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2448848896 bytes 2023-12-06T15:44:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2509188096 bytes 2023-12-06T15:44:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2575281152 bytes 2023-12-06T15:44:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2598748160 bytes 2023-12-06T15:44:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2606081024 bytes 2023-12-06T15:44:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2654070784 bytes 2023-12-06T15:44:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2731659264 bytes 2023-12-06T15:44:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2770756608 bytes 2023-12-06T15:44:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2842015744 bytes 2023-12-06T15:44:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2901521408 bytes 2023-12-06T15:44:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2942637056 bytes 2023-12-06T15:44:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2967242752 bytes 2023-12-06T15:44:44.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2982752256 bytes 2023-12-06T15:44:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 3094991872 bytes 2023-12-06T15:44:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6600 IOs, 3157444608 bytes 2023-12-06T15:44:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6700 IOs, 3181651968 bytes 2023-12-06T15:44:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6800 IOs, 3229105152 bytes 2023-12-06T15:44:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6900 IOs, 3261210624 bytes 2023-12-06T15:44:45.316 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4486694322936977) > weight (0.25) 2023-12-06T15:44:45.316 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9530104968399362) > weight (0.25) 2023-12-06T15:44:45.316 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.768846100703723) > weight (0.25) 2023-12-06T15:44:45.317 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9251958792370508) > weight (0.25) 2023-12-06T15:44:45.317 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2023-12-06T15:44:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7000 IOs, 3368859648 bytes 2023-12-06T15:44:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7100 IOs, 3415878656 bytes 2023-12-06T15:44:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7200 IOs, 3466345472 bytes 2023-12-06T15:44:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:44:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:44:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:44:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:44:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:44:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_8 2023-12-06T15:44:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-12-06T15:44:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_8 128M 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:44:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-12-06T15:44:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:44:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-12-06T15:44:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:44:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-06T15:44:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:44:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-06T15:44:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_8': 2023-12-06T15:44:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:44:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:44:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:44:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 128fab3bae5e 2023-12-06T15:44:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.128fab3bae5e 2023-12-06T15:44:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:44:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:44:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:44:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:44:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:45 2023 2023-12-06T15:44:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:45 2023 2023-12-06T15:44:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:45 2023 2023-12-06T15:44:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:44:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:44:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f4484446-8808-413e-bb2b-96fb61446c98 2023-12-06T15:44:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:44:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:44:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-06T15:44:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:44:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:44:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=11 2023-12-06T15:44:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:44:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:44:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 97482752 bytes 2023-12-06T15:44:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 122940416 bytes 2023-12-06T15:44:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 147562496 bytes 2023-12-06T15:44:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 241094656 bytes 2023-12-06T15:44:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 289698816 bytes 2023-12-06T15:44:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 374821888 bytes 2023-12-06T15:44:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 433348608 bytes 2023-12-06T15:44:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 459115520 bytes 2023-12-06T15:44:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 491487232 bytes 2023-12-06T15:44:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 577441792 bytes 2023-12-06T15:44:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 610363392 bytes 2023-12-06T15:44:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 651002880 bytes 2023-12-06T15:44:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 673805312 bytes 2023-12-06T15:44:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 709718016 bytes 2023-12-06T15:44:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 722178048 bytes 2023-12-06T15:44:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 804100096 bytes 2023-12-06T15:44:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 854220800 bytes 2023-12-06T15:44:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 861996032 bytes 2023-12-06T15:44:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 902831104 bytes 2023-12-06T15:44:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 951420928 bytes 2023-12-06T15:44:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 979106816 bytes 2023-12-06T15:44:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1006993408 bytes 2023-12-06T15:44:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1059186688 bytes 2023-12-06T15:44:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1118884864 bytes 2023-12-06T15:44:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1148345344 bytes 2023-12-06T15:44:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1208361984 bytes 2023-12-06T15:44:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1221200896 bytes 2023-12-06T15:44:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1269795840 bytes 2023-12-06T15:44:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1302129664 bytes 2023-12-06T15:44:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1335757824 bytes 2023-12-06T15:44:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1430492160 bytes 2023-12-06T15:44:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1504984064 bytes 2023-12-06T15:44:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1521861632 bytes 2023-12-06T15:44:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1561175040 bytes 2023-12-06T15:44:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1637911552 bytes 2023-12-06T15:44:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1681296384 bytes 2023-12-06T15:44:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1706301440 bytes 2023-12-06T15:44:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1750781952 bytes 2023-12-06T15:44:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1847841792 bytes 2023-12-06T15:44:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1894943744 bytes 2023-12-06T15:44:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1940903936 bytes 2023-12-06T15:44:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1944249344 bytes 2023-12-06T15:44:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 2012071936 bytes 2023-12-06T15:44:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 2036544512 bytes 2023-12-06T15:44:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 2086483968 bytes 2023-12-06T15:44:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 2103107584 bytes 2023-12-06T15:44:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2166495232 bytes 2023-12-06T15:44:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2169046016 bytes 2023-12-06T15:44:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2264489984 bytes 2023-12-06T15:44:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2288078848 bytes 2023-12-06T15:44:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2296607744 bytes 2023-12-06T15:44:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2314161152 bytes 2023-12-06T15:44:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2338587648 bytes 2023-12-06T15:44:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2380673024 bytes 2023-12-06T15:44:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2455714816 bytes 2023-12-06T15:44:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2533816320 bytes 2023-12-06T15:44:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2543293440 bytes 2023-12-06T15:44:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2599923712 bytes 2023-12-06T15:44:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2653153280 bytes 2023-12-06T15:44:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2671707136 bytes 2023-12-06T15:44:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:44:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:44:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:44:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:44:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:44:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_9 2023-12-06T15:44:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_9 128M 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:44:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:44:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:44:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:44:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:44:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2023-12-06T15:44:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:44:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-12-06T15:44:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:44:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:44:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-06T15:44:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:44:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-06T15:44:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_9': 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 129ad8725f83 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.129ad8725f83 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:57 2023 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:57 2023 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:57 2023 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:44:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:44:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f202fb70-81a7-49b9-b61d-6d5b735f390f 2023-12-06T15:44:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:44:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:44:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-06T15:44:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:44:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:44:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:44:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:44:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=6 2023-12-06T15:44:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:44:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:44:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 73541632 bytes 2023-12-06T15:44:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 152187904 bytes 2023-12-06T15:44:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 184264704 bytes 2023-12-06T15:44:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 254861312 bytes 2023-12-06T15:44:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 272163840 bytes 2023-12-06T15:44:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 367126528 bytes 2023-12-06T15:44:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 391001088 bytes 2023-12-06T15:44:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 462616576 bytes 2023-12-06T15:44:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 465700864 bytes 2023-12-06T15:44:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 481448960 bytes 2023-12-06T15:45:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 535153664 bytes 2023-12-06T15:45:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 589519872 bytes 2023-12-06T15:45:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 616648704 bytes 2023-12-06T15:45:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 663450624 bytes 2023-12-06T15:45:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 695477248 bytes 2023-12-06T15:45:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 764716032 bytes 2023-12-06T15:45:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 789732352 bytes 2023-12-06T15:45:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 823421952 bytes 2023-12-06T15:45:01.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 824060928 bytes 2023-12-06T15:45:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 874377216 bytes 2023-12-06T15:45:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 915824640 bytes 2023-12-06T15:45:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 955858944 bytes 2023-12-06T15:45:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 991887360 bytes 2023-12-06T15:45:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1052636160 bytes 2023-12-06T15:45:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1086879744 bytes 2023-12-06T15:45:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1124566016 bytes 2023-12-06T15:45:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1182126080 bytes 2023-12-06T15:45:02.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1222760448 bytes 2023-12-06T15:45:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1242777600 bytes 2023-12-06T15:45:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1246039040 bytes 2023-12-06T15:45:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1340424192 bytes 2023-12-06T15:45:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1388353536 bytes 2023-12-06T15:45:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1427036160 bytes 2023-12-06T15:45:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1448421376 bytes 2023-12-06T15:45:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:45:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:45:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:45:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:45:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:45:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_10 2023-12-06T15:45:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-12-06T15:45:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_10 128M 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:45:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2023-12-06T15:45:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:45:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-12-06T15:45:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:45:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:45:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-06T15:45:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:45:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-06T15:45:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_10': 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 12a24f3c0b2 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.12a24f3c0b2 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:45:03 2023 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:45:03 2023 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:45:03 2023 2023-12-06T15:45:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:45:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:45:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 70428fb0-ef2f-4a77-a9f3-f9c2d486736f 2023-12-06T15:45:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:45:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:45:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-06T15:45:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:45:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:45:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=7 2023-12-06T15:45:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:45:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:45:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 25602048 bytes 2023-12-06T15:45:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 41413632 bytes 2023-12-06T15:45:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 95793152 bytes 2023-12-06T15:45:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 102572032 bytes 2023-12-06T15:45:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 179667968 bytes 2023-12-06T15:45:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 209342464 bytes 2023-12-06T15:45:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 245668864 bytes 2023-12-06T15:45:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 305470464 bytes 2023-12-06T15:45:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 320163840 bytes 2023-12-06T15:45:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 351011840 bytes 2023-12-06T15:45:06.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 433434624 bytes 2023-12-06T15:45:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 473017344 bytes 2023-12-06T15:45:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 485257216 bytes 2023-12-06T15:45:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 506249216 bytes 2023-12-06T15:45:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 582532096 bytes 2023-12-06T15:45:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 606080000 bytes 2023-12-06T15:45:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 646733824 bytes 2023-12-06T15:45:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 700339200 bytes 2023-12-06T15:45:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 767385600 bytes 2023-12-06T15:45:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 792498176 bytes 2023-12-06T15:45:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 856393728 bytes 2023-12-06T15:45:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 926936064 bytes 2023-12-06T15:45:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 959735808 bytes 2023-12-06T15:45:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1037724672 bytes 2023-12-06T15:45:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1084816384 bytes 2023-12-06T15:45:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1105713152 bytes 2023-12-06T15:45:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1151293440 bytes 2023-12-06T15:45:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1178629120 bytes 2023-12-06T15:45:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1183544320 bytes 2023-12-06T15:45:10.224 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.40887031687586073) > weight (0.25) 2023-12-06T15:45:10.225 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7086679117464681) > weight (0.25) 2023-12-06T15:45:10.225 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9662102266934615) > weight (0.25) 2023-12-06T15:45:10.225 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5657533745625134) > weight (0.25) 2023-12-06T15:45:10.225 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2023-12-06T15:45:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1208028160 bytes 2023-12-06T15:45:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1253330944 bytes 2023-12-06T15:45:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1268591616 bytes 2023-12-06T15:45:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1281693696 bytes 2023-12-06T15:45:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1312040960 bytes 2023-12-06T15:45:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1369929728 bytes 2023-12-06T15:45:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1398000640 bytes 2023-12-06T15:45:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1438470144 bytes 2023-12-06T15:45:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1487988736 bytes 2023-12-06T15:45:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1543007232 bytes 2023-12-06T15:45:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1570666496 bytes 2023-12-06T15:45:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:45:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:45:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:45:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:45:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:45:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_11 2023-12-06T15:45:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-12-06T15:45:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_11 128M 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2023-12-06T15:45:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:45:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-12-06T15:45:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:45:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:45:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-12-06T15:45:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:45:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-12-06T15:45:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_11': 2023-12-06T15:45:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:45:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:45:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:45:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 12ade6889bfe 2023-12-06T15:45:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.12ade6889bfe 2023-12-06T15:45:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:45:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:45:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:45:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:45:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:45:12 2023 2023-12-06T15:45:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:45:12 2023 2023-12-06T15:45:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:45:12 2023 2023-12-06T15:45:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:45:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:45:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 85642e31-6be7-4259-b3a5-09a8ae17198f 2023-12-06T15:45:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:45:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:45:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_11 2023-12-06T15:45:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:45:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:45:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=11 2023-12-06T15:45:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:45:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:45:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 3475456 bytes 2023-12-06T15:45:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 81185792 bytes 2023-12-06T15:45:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 169016320 bytes 2023-12-06T15:45:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 194193408 bytes 2023-12-06T15:45:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 236262400 bytes 2023-12-06T15:45:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 246749184 bytes 2023-12-06T15:45:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 280623104 bytes 2023-12-06T15:45:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 331720704 bytes 2023-12-06T15:45:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 384507904 bytes 2023-12-06T15:45:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 416410624 bytes 2023-12-06T15:45:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 437799936 bytes 2023-12-06T15:45:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 488084480 bytes 2023-12-06T15:45:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 537000960 bytes 2023-12-06T15:45:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 578338816 bytes 2023-12-06T15:45:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 578971648 bytes 2023-12-06T15:45:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 617243648 bytes 2023-12-06T15:45:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 659779584 bytes 2023-12-06T15:45:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 764404736 bytes 2023-12-06T15:45:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 807913472 bytes 2023-12-06T15:45:16.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 896472064 bytes 2023-12-06T15:45:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 927154176 bytes 2023-12-06T15:45:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 957253632 bytes 2023-12-06T15:45:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 973870080 bytes 2023-12-06T15:45:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 984675328 bytes 2023-12-06T15:45:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1019927552 bytes 2023-12-06T15:45:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1031430144 bytes 2023-12-06T15:45:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1133114368 bytes 2023-12-06T15:45:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1185092608 bytes 2023-12-06T15:45:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1239561216 bytes 2023-12-06T15:45:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1284321280 bytes 2023-12-06T15:45:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1337574400 bytes 2023-12-06T15:45:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1444404224 bytes 2023-12-06T15:45:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1448849408 bytes 2023-12-06T15:45:19.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1501292544 bytes 2023-12-06T15:45:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1545903104 bytes 2023-12-06T15:45:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1567762432 bytes 2023-12-06T15:45:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1605101568 bytes 2023-12-06T15:45:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1670335488 bytes 2023-12-06T15:45:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1691784192 bytes 2023-12-06T15:45:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1747186688 bytes 2023-12-06T15:45:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1792446464 bytes 2023-12-06T15:45:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1821422592 bytes 2023-12-06T15:45:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1916090368 bytes 2023-12-06T15:45:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1979671552 bytes 2023-12-06T15:45:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1980288000 bytes 2023-12-06T15:45:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 2027612160 bytes 2023-12-06T15:45:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2051560448 bytes 2023-12-06T15:45:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2092720128 bytes 2023-12-06T15:45:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2131133440 bytes 2023-12-06T15:45:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2187897856 bytes 2023-12-06T15:45:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2233885696 bytes 2023-12-06T15:45:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2296358912 bytes 2023-12-06T15:45:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2304595968 bytes 2023-12-06T15:45:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2361703424 bytes 2023-12-06T15:45:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2386048000 bytes 2023-12-06T15:45:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2443557888 bytes 2023-12-06T15:45:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2471577600 bytes 2023-12-06T15:45:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2486003712 bytes 2023-12-06T15:45:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2526824448 bytes 2023-12-06T15:45:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2627300352 bytes 2023-12-06T15:45:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:45:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:45:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:45:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:45:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:45:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_12 2023-12-06T15:45:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-12-06T15:45:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:45:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:45:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:45:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_12 128M 2023-12-06T15:45:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:45:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:45:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:45:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:45:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2023-12-06T15:45:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:45:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-12-06T15:45:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:45:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:45:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-12-06T15:45:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:45:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-12-06T15:45:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_12': 2023-12-06T15:45:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 12b514d6304f 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.12b514d6304f 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:45:24 2023 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:45:24 2023 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:45:24 2023 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 2bd42a12-9068-4ea7-b5df-749817b7f6cc 2023-12-06T15:45:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:45:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:45:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_12 2023-12-06T15:45:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:45:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:45:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=9 2023-12-06T15:45:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:45:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:45:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 81599488 bytes 2023-12-06T15:45:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 123325440 bytes 2023-12-06T15:45:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 159766528 bytes 2023-12-06T15:45:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 189416448 bytes 2023-12-06T15:45:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 230487040 bytes 2023-12-06T15:45:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 284112896 bytes 2023-12-06T15:45:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 333487104 bytes 2023-12-06T15:45:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 416262144 bytes 2023-12-06T15:45:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 451886080 bytes 2023-12-06T15:45:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 452494336 bytes 2023-12-06T15:45:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 460615680 bytes 2023-12-06T15:45:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 475981824 bytes 2023-12-06T15:45:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 556954624 bytes 2023-12-06T15:45:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 655636480 bytes 2023-12-06T15:45:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 703747072 bytes 2023-12-06T15:45:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 754766848 bytes 2023-12-06T15:45:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 782472192 bytes 2023-12-06T15:45:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 813599744 bytes 2023-12-06T15:45:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 849044480 bytes 2023-12-06T15:45:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 876636160 bytes 2023-12-06T15:45:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 891538432 bytes 2023-12-06T15:45:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 924865536 bytes 2023-12-06T15:45:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 992612352 bytes 2023-12-06T15:45:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1090900992 bytes 2023-12-06T15:45:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1116493824 bytes 2023-12-06T15:45:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1180059648 bytes 2023-12-06T15:45:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1205056512 bytes 2023-12-06T15:45:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1229473792 bytes 2023-12-06T15:45:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1273049088 bytes 2023-12-06T15:45:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1298551808 bytes 2023-12-06T15:45:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1343431680 bytes 2023-12-06T15:45:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1365355520 bytes 2023-12-06T15:45:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1399123968 bytes 2023-12-06T15:45:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1447245824 bytes 2023-12-06T15:45:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1496582144 bytes 2023-12-06T15:45:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1604259840 bytes 2023-12-06T15:45:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1661585408 bytes 2023-12-06T15:45:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1695687680 bytes 2023-12-06T15:45:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1726871552 bytes 2023-12-06T15:45:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1739704320 bytes 2023-12-06T15:45:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1801782272 bytes 2023-12-06T15:45:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1844359168 bytes 2023-12-06T15:45:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1852473344 bytes 2023-12-06T15:45:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1873918976 bytes 2023-12-06T15:45:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1905332224 bytes 2023-12-06T15:45:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1954254848 bytes 2023-12-06T15:45:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2025196544 bytes 2023-12-06T15:45:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2131215360 bytes 2023-12-06T15:45:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_13 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:45:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:45:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_13 128M 2023-12-06T15:45:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:45:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:45:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:45:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:45:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2023-12-06T15:45:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:45:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-12-06T15:45:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:45:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:45:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-12-06T15:45:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:45:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-12-06T15:45:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_13': 2023-12-06T15:45:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:45:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:45:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:45:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 12bfa4c18acd 2023-12-06T15:45:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.12bfa4c18acd 2023-12-06T15:45:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:45:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:45:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:45:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:45:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:45:34 2023 2023-12-06T15:45:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:45:34 2023 2023-12-06T15:45:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:45:34 2023 2023-12-06T15:45:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:45:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:45:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: d04aac8e-3190-4279-b4cb-4b1a37a6b42f 2023-12-06T15:45:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:45:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:45:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_13 2023-12-06T15:45:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:45:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:45:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=14 2023-12-06T15:45:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:45:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:45:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 32327680 bytes 2023-12-06T15:45:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 59600896 bytes 2023-12-06T15:45:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 82756608 bytes 2023-12-06T15:45:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 223410176 bytes 2023-12-06T15:45:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 283700224 bytes 2023-12-06T15:45:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 325950464 bytes 2023-12-06T15:45:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 408655872 bytes 2023-12-06T15:45:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 451068928 bytes 2023-12-06T15:45:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 480430080 bytes 2023-12-06T15:45:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 504867840 bytes 2023-12-06T15:45:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 560071680 bytes 2023-12-06T15:45:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 594422784 bytes 2023-12-06T15:45:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 670995456 bytes 2023-12-06T15:45:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 723223552 bytes 2023-12-06T15:45:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 774549504 bytes 2023-12-06T15:45:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 808604672 bytes 2023-12-06T15:45:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 841885696 bytes 2023-12-06T15:45:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 878074880 bytes 2023-12-06T15:45:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 929300480 bytes 2023-12-06T15:45:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 975971328 bytes 2023-12-06T15:45:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 1040791552 bytes 2023-12-06T15:45:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1076486144 bytes 2023-12-06T15:45:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1077112832 bytes 2023-12-06T15:45:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1100535808 bytes 2023-12-06T15:45:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1123500032 bytes 2023-12-06T15:45:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1184441344 bytes 2023-12-06T15:45:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1256346624 bytes 2023-12-06T15:45:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1291680768 bytes 2023-12-06T15:45:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1293920256 bytes 2023-12-06T15:45:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1335254016 bytes 2023-12-06T15:45:40.666 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:45:40.664+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8ece79e10 2023-12-06T15:45:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1412671488 bytes 2023-12-06T15:45:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1481176064 bytes 2023-12-06T15:45:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1506012160 bytes 2023-12-06T15:45:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1541106688 bytes 2023-12-06T15:45:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1579955200 bytes 2023-12-06T15:45:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1636149248 bytes 2023-12-06T15:45:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1674381312 bytes 2023-12-06T15:45:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1700291584 bytes 2023-12-06T15:45:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1765847040 bytes 2023-12-06T15:45:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1793024000 bytes 2023-12-06T15:45:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1811562496 bytes 2023-12-06T15:45:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1893134336 bytes 2023-12-06T15:45:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1930900480 bytes 2023-12-06T15:45:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1966949376 bytes 2023-12-06T15:45:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 2013080576 bytes 2023-12-06T15:45:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 2041705472 bytes 2023-12-06T15:45:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2106139648 bytes 2023-12-06T15:45:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2154128384 bytes 2023-12-06T15:45:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2198558720 bytes 2023-12-06T15:45:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2213121024 bytes 2023-12-06T15:45:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2279317504 bytes 2023-12-06T15:45:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2334755840 bytes 2023-12-06T15:45:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2374927360 bytes 2023-12-06T15:45:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2426305536 bytes 2023-12-06T15:45:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2493808640 bytes 2023-12-06T15:45:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2526052352 bytes 2023-12-06T15:45:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2605255680 bytes 2023-12-06T15:45:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2629538816 bytes 2023-12-06T15:45:45.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2648443904 bytes 2023-12-06T15:45:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2649053184 bytes 2023-12-06T15:45:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2701871104 bytes 2023-12-06T15:45:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2721568768 bytes 2023-12-06T15:45:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2828613632 bytes 2023-12-06T15:45:46.751 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:45:46.747+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8ece79e10 2023-12-06T15:45:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2845347840 bytes 2023-12-06T15:45:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 2886354944 bytes 2023-12-06T15:45:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6600 IOs, 3031374848 bytes 2023-12-06T15:45:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6700 IOs, 3161511936 bytes 2023-12-06T15:45:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6800 IOs, 3200364544 bytes 2023-12-06T15:45:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6900 IOs, 3238370304 bytes 2023-12-06T15:45:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7000 IOs, 3324084224 bytes 2023-12-06T15:45:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7100 IOs, 3366360064 bytes 2023-12-06T15:45:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7200 IOs, 3440953344 bytes 2023-12-06T15:45:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7300 IOs, 3451921408 bytes 2023-12-06T15:45:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_14 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_14 128M 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2023-12-06T15:45:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:45:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-12-06T15:45:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:45:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:45:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-12-06T15:45:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:45:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-12-06T15:45:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_14': 2023-12-06T15:45:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 12c7b3f05918 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.12c7b3f05918 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:45:49 2023 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:45:49 2023 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:45:49 2023 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:45:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 216c840f-8878-4153-b225-e1d08de4c3fd 2023-12-06T15:45:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:45:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:45:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_14 2023-12-06T15:45:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:45:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:45:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=8 2023-12-06T15:45:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:45:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:45:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 38578176 bytes 2023-12-06T15:45:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 124476416 bytes 2023-12-06T15:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 212793344 bytes 2023-12-06T15:45:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 238539776 bytes 2023-12-06T15:45:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 316919808 bytes 2023-12-06T15:45:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 351600640 bytes 2023-12-06T15:45:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 384588800 bytes 2023-12-06T15:45:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 411331584 bytes 2023-12-06T15:45:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 455697408 bytes 2023-12-06T15:45:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 490892288 bytes 2023-12-06T15:45:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 541424640 bytes 2023-12-06T15:45:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 574472192 bytes 2023-12-06T15:45:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 595977216 bytes 2023-12-06T15:45:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 614795264 bytes 2023-12-06T15:45:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 663120896 bytes 2023-12-06T15:45:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 724628480 bytes 2023-12-06T15:45:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 725239808 bytes 2023-12-06T15:45:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 776064000 bytes 2023-12-06T15:45:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 912800768 bytes 2023-12-06T15:45:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 925791232 bytes 2023-12-06T15:45:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 944025600 bytes 2023-12-06T15:45:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 970944512 bytes 2023-12-06T15:45:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1093297152 bytes 2023-12-06T15:45:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1144279040 bytes 2023-12-06T15:45:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1175941120 bytes 2023-12-06T15:45:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1257424896 bytes 2023-12-06T15:45:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1291022336 bytes 2023-12-06T15:45:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1322138624 bytes 2023-12-06T15:45:55.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1365355520 bytes 2023-12-06T15:45:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1440515072 bytes 2023-12-06T15:45:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1503195136 bytes 2023-12-06T15:45:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1573156864 bytes 2023-12-06T15:45:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1627058176 bytes 2023-12-06T15:45:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1693956096 bytes 2023-12-06T15:45:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1704671232 bytes 2023-12-06T15:45:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1740759040 bytes 2023-12-06T15:45:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1796620288 bytes 2023-12-06T15:45:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1847431168 bytes 2023-12-06T15:45:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1929842688 bytes 2023-12-06T15:45:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1945817088 bytes 2023-12-06T15:45:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1987358720 bytes 2023-12-06T15:45:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:45:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:45:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:45:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:45:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:45:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_15 2023-12-06T15:45:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-12-06T15:45:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_15 128M 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:45:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2023-12-06T15:45:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:45:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-12-06T15:45:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:45:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:45:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-12-06T15:45:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:45:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-12-06T15:45:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_15': 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 12d0f53a41c2 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.12d0f53a41c2 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:45:58 2023 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:45:58 2023 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:45:58 2023 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:45:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 500b919a-f976-4b7b-8c44-5adea4cfc7db 2023-12-06T15:45:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:45:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:45:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_15 2023-12-06T15:45:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:45:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:45:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:45:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:45:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=14 2023-12-06T15:45:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:45:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:45:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 93458432 bytes 2023-12-06T15:45:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 179618816 bytes 2023-12-06T15:45:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 246940672 bytes 2023-12-06T15:45:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 312114176 bytes 2023-12-06T15:46:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 342097920 bytes 2023-12-06T15:46:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 392507392 bytes 2023-12-06T15:46:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 460364800 bytes 2023-12-06T15:46:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 495522816 bytes 2023-12-06T15:46:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 517878784 bytes 2023-12-06T15:46:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 563437568 bytes 2023-12-06T15:46:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 612744192 bytes 2023-12-06T15:46:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 672262144 bytes 2023-12-06T15:46:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 704065536 bytes 2023-12-06T15:46:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 745436160 bytes 2023-12-06T15:46:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 801244160 bytes 2023-12-06T15:46:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 864156672 bytes 2023-12-06T15:46:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 878490624 bytes 2023-12-06T15:46:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 930279424 bytes 2023-12-06T15:46:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 940575744 bytes 2023-12-06T15:46:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 969465856 bytes 2023-12-06T15:46:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 1012562944 bytes 2023-12-06T15:46:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1043560448 bytes 2023-12-06T15:46:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1122127872 bytes 2023-12-06T15:46:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1131936768 bytes 2023-12-06T15:46:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1182277632 bytes 2023-12-06T15:46:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1192900608 bytes 2023-12-06T15:46:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1280052224 bytes 2023-12-06T15:46:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1348894720 bytes 2023-12-06T15:46:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1356206080 bytes 2023-12-06T15:46:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1436286976 bytes 2023-12-06T15:46:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1479626752 bytes 2023-12-06T15:46:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1609854976 bytes 2023-12-06T15:46:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1633773568 bytes 2023-12-06T15:46:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1717611520 bytes 2023-12-06T15:46:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1762361344 bytes 2023-12-06T15:46:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1764466688 bytes 2023-12-06T15:46:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1788363776 bytes 2023-12-06T15:46:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1790581760 bytes 2023-12-06T15:46:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1824089088 bytes 2023-12-06T15:46:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1902831616 bytes 2023-12-06T15:46:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1941839872 bytes 2023-12-06T15:46:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 2019155968 bytes 2023-12-06T15:46:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 2037061632 bytes 2023-12-06T15:46:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 2058769408 bytes 2023-12-06T15:46:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 2114804736 bytes 2023-12-06T15:46:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 2142977024 bytes 2023-12-06T15:46:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2179949568 bytes 2023-12-06T15:46:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2210969600 bytes 2023-12-06T15:46:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2230326272 bytes 2023-12-06T15:46:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2290975744 bytes 2023-12-06T15:46:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2327739392 bytes 2023-12-06T15:46:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2432712704 bytes 2023-12-06T15:46:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2433345536 bytes 2023-12-06T15:46:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2478306304 bytes 2023-12-06T15:46:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2547762176 bytes 2023-12-06T15:46:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2631617536 bytes 2023-12-06T15:46:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2669332480 bytes 2023-12-06T15:46:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2700126208 bytes 2023-12-06T15:46:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2764323840 bytes 2023-12-06T15:46:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2791102464 bytes 2023-12-06T15:46:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2842147840 bytes 2023-12-06T15:46:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2891106304 bytes 2023-12-06T15:46:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2961995776 bytes 2023-12-06T15:46:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2994653184 bytes 2023-12-06T15:46:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 3104416768 bytes 2023-12-06T15:46:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6600 IOs, 3123736576 bytes 2023-12-06T15:46:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6700 IOs, 3190782976 bytes 2023-12-06T15:46:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6800 IOs, 3205991424 bytes 2023-12-06T15:46:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6900 IOs, 3238034432 bytes 2023-12-06T15:46:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7000 IOs, 3303977984 bytes 2023-12-06T15:46:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7100 IOs, 3353109504 bytes 2023-12-06T15:46:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7200 IOs, 3369875456 bytes 2023-12-06T15:46:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7300 IOs, 3418571776 bytes 2023-12-06T15:46:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7400 IOs, 3450423296 bytes 2023-12-06T15:46:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:46:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:46:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:46:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:46:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:46:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_16 2023-12-06T15:46:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-12-06T15:46:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_16 128M 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:46:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:46:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:46:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:46:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2023-12-06T15:46:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:46:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-12-06T15:46:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:46:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:46:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-12-06T15:46:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:46:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-12-06T15:46:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_16': 2023-12-06T15:46:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:46:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:46:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:46:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 12dbbd9aaa8e 2023-12-06T15:46:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.12dbbd9aaa8e 2023-12-06T15:46:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:46:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:46:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:46:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:46:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:46:12 2023 2023-12-06T15:46:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:46:12 2023 2023-12-06T15:46:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:46:12 2023 2023-12-06T15:46:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:46:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:46:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: dc7f2a11-16c4-4259-8d8f-2015f551ea93 2023-12-06T15:46:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:46:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:46:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_16 2023-12-06T15:46:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:46:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:46:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=10 2023-12-06T15:46:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:46:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:46:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 79686656 bytes 2023-12-06T15:46:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 96533504 bytes 2023-12-06T15:46:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 172667904 bytes 2023-12-06T15:46:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 211293184 bytes 2023-12-06T15:46:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 237131776 bytes 2023-12-06T15:46:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 282252288 bytes 2023-12-06T15:46:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 329659392 bytes 2023-12-06T15:46:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 369310720 bytes 2023-12-06T15:46:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 422838272 bytes 2023-12-06T15:46:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 492135424 bytes 2023-12-06T15:46:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 524617728 bytes 2023-12-06T15:46:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 546914304 bytes 2023-12-06T15:46:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 610674688 bytes 2023-12-06T15:46:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 644485120 bytes 2023-12-06T15:46:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 650510336 bytes 2023-12-06T15:46:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 678145024 bytes 2023-12-06T15:46:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 719680512 bytes 2023-12-06T15:46:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 759629824 bytes 2023-12-06T15:46:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 819173376 bytes 2023-12-06T15:46:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 868296704 bytes 2023-12-06T15:46:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 929807360 bytes 2023-12-06T15:46:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1003384832 bytes 2023-12-06T15:46:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1023959040 bytes 2023-12-06T15:46:18.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1122041856 bytes 2023-12-06T15:46:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1149122560 bytes 2023-12-06T15:46:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1189378048 bytes 2023-12-06T15:46:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1242591232 bytes 2023-12-06T15:46:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1300663296 bytes 2023-12-06T15:46:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1362460672 bytes 2023-12-06T15:46:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1421477888 bytes 2023-12-06T15:46:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1491373056 bytes 2023-12-06T15:46:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1543054336 bytes 2023-12-06T15:46:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1555613696 bytes 2023-12-06T15:46:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1569846272 bytes 2023-12-06T15:46:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1643744256 bytes 2023-12-06T15:46:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1702639616 bytes 2023-12-06T15:46:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1705476096 bytes 2023-12-06T15:46:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1751144448 bytes 2023-12-06T15:46:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1848184832 bytes 2023-12-06T15:46:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1874647040 bytes 2023-12-06T15:46:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1926728704 bytes 2023-12-06T15:46:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1927336960 bytes 2023-12-06T15:46:22.226 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.27637959192701034) > weight (0.25) 2023-12-06T15:46:22.226 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.580065400218201) > weight (0.25) 2023-12-06T15:46:22.226 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-06T15:46:22.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-06T15:46:22.227 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-06T15:46:22.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T15:46:22.226+0000 7fdfdc745700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 123314) UID: 1000 2023-12-06T15:46:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1927968768 bytes 2023-12-06T15:46:22.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1956450304 bytes 2023-12-06T15:46:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1969974272 bytes 2023-12-06T15:46:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 2059661312 bytes 2023-12-06T15:46:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2060290048 bytes 2023-12-06T15:46:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2060897280 bytes 2023-12-06T15:46:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2076197888 bytes 2023-12-06T15:46:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2101072896 bytes 2023-12-06T15:46:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2140871680 bytes 2023-12-06T15:46:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2154642432 bytes 2023-12-06T15:46:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_17 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_17 128M 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2023-12-06T15:46:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:46:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-12-06T15:46:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:46:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:46:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-12-06T15:46:24.318 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5896362609496275) > weight (0.25) 2023-12-06T15:46:24.318 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-06T15:46:24.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-06T15:46:24.318 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-06T15:46:24.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T15:46:24.317+0000 7f5e0f6f1700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 119429) UID: 1000 2023-12-06T15:46:24.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:46:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:46:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-12-06T15:46:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_17': 2023-12-06T15:46:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:46:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:46:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:46:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 12e6e4188b50 2023-12-06T15:46:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.12e6e4188b50 2023-12-06T15:46:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:46:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:46:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:46:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:46:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:46:24 2023 2023-12-06T15:46:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:46:24 2023 2023-12-06T15:46:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:46:24 2023 2023-12-06T15:46:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:46:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:46:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: b11d8c4b-991a-40fd-a328-c851d1d3afd6 2023-12-06T15:46:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:46:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:46:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_17 2023-12-06T15:46:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:46:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:46:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=6 2023-12-06T15:46:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:46:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:46:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 112183296 bytes 2023-12-06T15:46:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 222688256 bytes 2023-12-06T15:46:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 277281792 bytes 2023-12-06T15:46:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 406579200 bytes 2023-12-06T15:46:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 430789632 bytes 2023-12-06T15:46:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 481708032 bytes 2023-12-06T15:46:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 530623488 bytes 2023-12-06T15:46:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 540960768 bytes 2023-12-06T15:46:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 582831104 bytes 2023-12-06T15:46:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 619293696 bytes 2023-12-06T15:46:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 638862336 bytes 2023-12-06T15:46:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 701569024 bytes 2023-12-06T15:46:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 784359424 bytes 2023-12-06T15:46:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 801724416 bytes 2023-12-06T15:46:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 820496384 bytes 2023-12-06T15:46:28.320 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-06T15:46:28.320 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:46:28.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:46:28.320 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-06T15:46:28.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-06T15:46:28.321 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-06T15:46:28.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-06T15:46:28.323 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2023-12-06T15:46:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 884707328 bytes 2023-12-06T15:46:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 941663232 bytes 2023-12-06T15:46:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 983889920 bytes 2023-12-06T15:46:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 1060445184 bytes 2023-12-06T15:46:29.228 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-06T15:46:29.228 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:46:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:46:29.228 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-06T15:46:29.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-06T15:46:29.229 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-06T15:46:29.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-06T15:46:29.231 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2023-12-06T15:46:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 1112297472 bytes 2023-12-06T15:46:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 1201802240 bytes 2023-12-06T15:46:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1277529088 bytes 2023-12-06T15:46:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1281238016 bytes 2023-12-06T15:46:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1300704256 bytes 2023-12-06T15:46:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1316481024 bytes 2023-12-06T15:46:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1382321152 bytes 2023-12-06T15:46:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1427948544 bytes 2023-12-06T15:46:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1436292096 bytes 2023-12-06T15:46:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1483592704 bytes 2023-12-06T15:46:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:46:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:46:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:46:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:46:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:46:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_18 2023-12-06T15:46:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_18 128M 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:46:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:46:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:46:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:46:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2023-12-06T15:46:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:46:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-12-06T15:46:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:46:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:46:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-12-06T15:46:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:46:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-12-06T15:46:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_18': 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 12ffbcb7bc96 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.12ffbcb7bc96 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:46:31 2023 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:46:31 2023 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:46:31 2023 2023-12-06T15:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:46:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:46:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: bf968f6f-af59-4018-a2bf-c1b130c0104d 2023-12-06T15:46:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:46:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:46:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_18 2023-12-06T15:46:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:46:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:46:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=11 2023-12-06T15:46:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:46:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:46:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 33259520 bytes 2023-12-06T15:46:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 44768256 bytes 2023-12-06T15:46:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 78570496 bytes 2023-12-06T15:46:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 140077056 bytes 2023-12-06T15:46:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 167244800 bytes 2023-12-06T15:46:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 177896448 bytes 2023-12-06T15:46:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 241440768 bytes 2023-12-06T15:46:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 290563072 bytes 2023-12-06T15:46:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 317825024 bytes 2023-12-06T15:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 370507776 bytes 2023-12-06T15:46:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 470405120 bytes 2023-12-06T15:46:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 505398272 bytes 2023-12-06T15:46:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 538657792 bytes 2023-12-06T15:46:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 583656448 bytes 2023-12-06T15:46:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 651816960 bytes 2023-12-06T15:46:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 722694144 bytes 2023-12-06T15:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 747550720 bytes 2023-12-06T15:46:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 750393344 bytes 2023-12-06T15:46:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 784497664 bytes 2023-12-06T15:46:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 836703232 bytes 2023-12-06T15:46:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 845863936 bytes 2023-12-06T15:46:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 867545088 bytes 2023-12-06T15:46:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 875759616 bytes 2023-12-06T15:46:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 893000704 bytes 2023-12-06T15:46:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 940640256 bytes 2023-12-06T15:46:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1016120320 bytes 2023-12-06T15:46:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1062608896 bytes 2023-12-06T15:46:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1084751872 bytes 2023-12-06T15:46:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1138026496 bytes 2023-12-06T15:46:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1160830976 bytes 2023-12-06T15:46:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1229337600 bytes 2023-12-06T15:46:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1248124928 bytes 2023-12-06T15:46:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1275084800 bytes 2023-12-06T15:46:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1363231744 bytes 2023-12-06T15:46:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1392090112 bytes 2023-12-06T15:46:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1392708608 bytes 2023-12-06T15:46:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1452683264 bytes 2023-12-06T15:46:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1503752192 bytes 2023-12-06T15:46:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1538681856 bytes 2023-12-06T15:46:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1574025216 bytes 2023-12-06T15:46:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1630872576 bytes 2023-12-06T15:46:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1700393984 bytes 2023-12-06T15:46:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1716575232 bytes 2023-12-06T15:46:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1761115136 bytes 2023-12-06T15:46:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1802726400 bytes 2023-12-06T15:46:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1886872576 bytes 2023-12-06T15:46:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1977458688 bytes 2023-12-06T15:46:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2065234944 bytes 2023-12-06T15:46:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2127023104 bytes 2023-12-06T15:46:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2145397760 bytes 2023-12-06T15:46:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2198312960 bytes 2023-12-06T15:46:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2271961088 bytes 2023-12-06T15:46:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2321821696 bytes 2023-12-06T15:46:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2325606400 bytes 2023-12-06T15:46:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2350387200 bytes 2023-12-06T15:46:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2356144128 bytes 2023-12-06T15:46:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2381123584 bytes 2023-12-06T15:46:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2461057024 bytes 2023-12-06T15:46:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2492963840 bytes 2023-12-06T15:46:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2518316032 bytes 2023-12-06T15:46:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:46:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:46:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:46:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:46:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_19 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_19 128M 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:46:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:46:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:46:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:46:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2023-12-06T15:46:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:46:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-12-06T15:46:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:46:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:46:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-12-06T15:46:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:46:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-12-06T15:46:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_19': 2023-12-06T15:46:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:46:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:46:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:46:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 130b9bfea5b2 2023-12-06T15:46:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.130b9bfea5b2 2023-12-06T15:46:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:46:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:46:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:46:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:46:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:46:43 2023 2023-12-06T15:46:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:46:43 2023 2023-12-06T15:46:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:46:43 2023 2023-12-06T15:46:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:46:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:46:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: cadad124-8d2e-48fe-a7b8-df7c14a3f2c0 2023-12-06T15:46:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:46:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:46:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_19 2023-12-06T15:46:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:46:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:46:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=6 2023-12-06T15:46:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:46:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:46:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 29176832 bytes 2023-12-06T15:46:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 62044160 bytes 2023-12-06T15:46:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 124261376 bytes 2023-12-06T15:46:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 124881920 bytes 2023-12-06T15:46:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 168026112 bytes 2023-12-06T15:46:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 241920000 bytes 2023-12-06T15:46:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 289419264 bytes 2023-12-06T15:46:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 323092480 bytes 2023-12-06T15:46:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 365324288 bytes 2023-12-06T15:46:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 442256384 bytes 2023-12-06T15:46:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 540259328 bytes 2023-12-06T15:46:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 592596992 bytes 2023-12-06T15:46:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 668138496 bytes 2023-12-06T15:46:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 706306048 bytes 2023-12-06T15:46:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 740513792 bytes 2023-12-06T15:46:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 784672768 bytes 2023-12-06T15:46:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 829575168 bytes 2023-12-06T15:46:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 877110272 bytes 2023-12-06T15:46:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 881367040 bytes 2023-12-06T15:46:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 881998848 bytes 2023-12-06T15:46:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 951673856 bytes 2023-12-06T15:46:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1003314176 bytes 2023-12-06T15:46:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1027181568 bytes 2023-12-06T15:46:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1109516288 bytes 2023-12-06T15:46:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1132872704 bytes 2023-12-06T15:46:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1207162880 bytes 2023-12-06T15:46:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1208561664 bytes 2023-12-06T15:46:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1231397888 bytes 2023-12-06T15:46:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1257344000 bytes 2023-12-06T15:46:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1295050752 bytes 2023-12-06T15:46:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1314755584 bytes 2023-12-06T15:46:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:46:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_20 2023-12-06T15:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-12-06T15:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:46:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:46:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:46:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_20 128M 2023-12-06T15:46:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:46:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:46:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:46:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:46:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:46:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2023-12-06T15:46:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:46:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-12-06T15:46:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:46:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:46:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-12-06T15:46:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:46:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-12-06T15:46:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_20': 2023-12-06T15:46:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 1317c92a43ba 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.1317c92a43ba 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:46:50 2023 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:46:50 2023 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:46:50 2023 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:46:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:46:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 7c1ca29b-6451-4600-b10e-63d11596db15 2023-12-06T15:46:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:46:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:46:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_20 2023-12-06T15:46:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:46:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:46:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:46:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:46:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=12 2023-12-06T15:46:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:46:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:46:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 75384832 bytes 2023-12-06T15:46:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 96550912 bytes 2023-12-06T15:46:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 174417920 bytes 2023-12-06T15:46:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 195367936 bytes 2023-12-06T15:46:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 208183296 bytes 2023-12-06T15:46:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 273997824 bytes 2023-12-06T15:46:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 317721600 bytes 2023-12-06T15:46:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 346472448 bytes 2023-12-06T15:46:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 395865088 bytes 2023-12-06T15:46:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 421567488 bytes 2023-12-06T15:46:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 476533760 bytes 2023-12-06T15:46:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 538833920 bytes 2023-12-06T15:46:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 584264704 bytes 2023-12-06T15:46:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 638543872 bytes 2023-12-06T15:46:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 661903360 bytes 2023-12-06T15:46:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 678621184 bytes 2023-12-06T15:46:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 700983296 bytes 2023-12-06T15:46:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 711136256 bytes 2023-12-06T15:46:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 734992384 bytes 2023-12-06T15:46:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 785107968 bytes 2023-12-06T15:46:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 831627264 bytes 2023-12-06T15:46:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 902691840 bytes 2023-12-06T15:46:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 935584768 bytes 2023-12-06T15:46:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1037300736 bytes 2023-12-06T15:46:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1146984448 bytes 2023-12-06T15:46:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1178245120 bytes 2023-12-06T15:46:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1180070912 bytes 2023-12-06T15:46:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1219068928 bytes 2023-12-06T15:46:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1285514240 bytes 2023-12-06T15:46:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1316015104 bytes 2023-12-06T15:46:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1322596352 bytes 2023-12-06T15:46:57.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1414380544 bytes 2023-12-06T15:46:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1450975232 bytes 2023-12-06T15:46:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1521475584 bytes 2023-12-06T15:46:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1534788608 bytes 2023-12-06T15:46:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1580794880 bytes 2023-12-06T15:46:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1652278272 bytes 2023-12-06T15:46:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1687154688 bytes 2023-12-06T15:46:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1702157312 bytes 2023-12-06T15:46:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1783396352 bytes 2023-12-06T15:46:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1849555968 bytes 2023-12-06T15:46:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1913292800 bytes 2023-12-06T15:46:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1948963840 bytes 2023-12-06T15:46:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 2001456128 bytes 2023-12-06T15:46:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 2033077248 bytes 2023-12-06T15:47:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 2076838912 bytes 2023-12-06T15:47:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2091155456 bytes 2023-12-06T15:47:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2143906816 bytes 2023-12-06T15:47:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2189453312 bytes 2023-12-06T15:47:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2230417408 bytes 2023-12-06T15:47:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2247140352 bytes 2023-12-06T15:47:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2281526272 bytes 2023-12-06T15:47:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2310746112 bytes 2023-12-06T15:47:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2329168896 bytes 2023-12-06T15:47:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2329778176 bytes 2023-12-06T15:47:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2333527040 bytes 2023-12-06T15:47:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2352953344 bytes 2023-12-06T15:47:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2394823680 bytes 2023-12-06T15:47:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2423865344 bytes 2023-12-06T15:47:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2475063296 bytes 2023-12-06T15:47:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2529754112 bytes 2023-12-06T15:47:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2557320192 bytes 2023-12-06T15:47:02.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2585793536 bytes 2023-12-06T15:47:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:47:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:47:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:47:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:47:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:47:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_21 2023-12-06T15:47:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_21 128M 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:47:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:47:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:47:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:47:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2023-12-06T15:47:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:47:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-12-06T15:47:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:47:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-12-06T15:47:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:47:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-12-06T15:47:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_21': 2023-12-06T15:47:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:47:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:47:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:47:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 1323de1c5846 2023-12-06T15:47:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.1323de1c5846 2023-12-06T15:47:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:47:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:47:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:47:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:47:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:03 2023 2023-12-06T15:47:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:03 2023 2023-12-06T15:47:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:03 2023 2023-12-06T15:47:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:47:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:47:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 298c48a0-cc50-45ca-995e-2df71ee446f2 2023-12-06T15:47:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:47:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:47:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_21 2023-12-06T15:47:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:47:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:47:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=11 2023-12-06T15:47:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:47:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:47:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 46082048 bytes 2023-12-06T15:47:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 64937984 bytes 2023-12-06T15:47:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 107884544 bytes 2023-12-06T15:47:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 129958912 bytes 2023-12-06T15:47:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 191989760 bytes 2023-12-06T15:47:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 243206144 bytes 2023-12-06T15:47:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 244973568 bytes 2023-12-06T15:47:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 289648640 bytes 2023-12-06T15:47:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 307675136 bytes 2023-12-06T15:47:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 319894528 bytes 2023-12-06T15:47:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 374877184 bytes 2023-12-06T15:47:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 452716544 bytes 2023-12-06T15:47:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 503717888 bytes 2023-12-06T15:47:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 511605760 bytes 2023-12-06T15:47:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 520902656 bytes 2023-12-06T15:47:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 521522176 bytes 2023-12-06T15:47:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 552865792 bytes 2023-12-06T15:47:07.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 596121600 bytes 2023-12-06T15:47:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 665479168 bytes 2023-12-06T15:47:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 675200000 bytes 2023-12-06T15:47:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 733148160 bytes 2023-12-06T15:47:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 787382272 bytes 2023-12-06T15:47:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 835396608 bytes 2023-12-06T15:47:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 861177856 bytes 2023-12-06T15:47:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 932539392 bytes 2023-12-06T15:47:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 985741312 bytes 2023-12-06T15:47:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1009802240 bytes 2023-12-06T15:47:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1092601856 bytes 2023-12-06T15:47:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1146650624 bytes 2023-12-06T15:47:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1201959936 bytes 2023-12-06T15:47:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1231486976 bytes 2023-12-06T15:47:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1272073216 bytes 2023-12-06T15:47:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1319195648 bytes 2023-12-06T15:47:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1370233856 bytes 2023-12-06T15:47:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1420861440 bytes 2023-12-06T15:47:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1421480960 bytes 2023-12-06T15:47:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1434168320 bytes 2023-12-06T15:47:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1534950400 bytes 2023-12-06T15:47:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1565452288 bytes 2023-12-06T15:47:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1593140224 bytes 2023-12-06T15:47:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1593787392 bytes 2023-12-06T15:47:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1594404864 bytes 2023-12-06T15:47:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1631289344 bytes 2023-12-06T15:47:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1668396032 bytes 2023-12-06T15:47:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1684384768 bytes 2023-12-06T15:47:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1756867584 bytes 2023-12-06T15:47:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1786797056 bytes 2023-12-06T15:47:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1879300096 bytes 2023-12-06T15:47:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1925841920 bytes 2023-12-06T15:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1944785920 bytes 2023-12-06T15:47:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 1969538048 bytes 2023-12-06T15:47:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 1979968512 bytes 2023-12-06T15:47:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2071789568 bytes 2023-12-06T15:47:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2101648384 bytes 2023-12-06T15:47:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2132849664 bytes 2023-12-06T15:47:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2202550272 bytes 2023-12-06T15:47:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2229453824 bytes 2023-12-06T15:47:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2255919104 bytes 2023-12-06T15:47:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2317620224 bytes 2023-12-06T15:47:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2371077120 bytes 2023-12-06T15:47:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2371707904 bytes 2023-12-06T15:47:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:47:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:47:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:47:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:47:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:47:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_22 2023-12-06T15:47:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-12-06T15:47:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_22 128M 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:47:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:47:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2023-12-06T15:47:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:47:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-12-06T15:47:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:47:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-12-06T15:47:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:47:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-12-06T15:47:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_22': 2023-12-06T15:47:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:47:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 132fa87ef9 2023-12-06T15:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.132fa87ef9 2023-12-06T15:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:15 2023 2023-12-06T15:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:15 2023 2023-12-06T15:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:15 2023 2023-12-06T15:47:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:47:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:47:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f9078af2-0a9f-4ece-8c12-87bbc54a7d15 2023-12-06T15:47:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:47:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:47:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_22 2023-12-06T15:47:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:47:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:47:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=6 2023-12-06T15:47:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:47:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:47:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 150691840 bytes 2023-12-06T15:47:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 186268672 bytes 2023-12-06T15:47:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 206628864 bytes 2023-12-06T15:47:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 227138560 bytes 2023-12-06T15:47:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 231187456 bytes 2023-12-06T15:47:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 254170112 bytes 2023-12-06T15:47:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 314023936 bytes 2023-12-06T15:47:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 377153536 bytes 2023-12-06T15:47:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 415860736 bytes 2023-12-06T15:47:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 434413568 bytes 2023-12-06T15:47:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 487489536 bytes 2023-12-06T15:47:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 508443648 bytes 2023-12-06T15:47:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 543209472 bytes 2023-12-06T15:47:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 601657344 bytes 2023-12-06T15:47:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 648489984 bytes 2023-12-06T15:47:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 693664768 bytes 2023-12-06T15:47:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 717450240 bytes 2023-12-06T15:47:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 777639936 bytes 2023-12-06T15:47:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 894080000 bytes 2023-12-06T15:47:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 919861248 bytes 2023-12-06T15:47:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 1002676224 bytes 2023-12-06T15:47:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1047178240 bytes 2023-12-06T15:47:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1073387520 bytes 2023-12-06T15:47:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1121405952 bytes 2023-12-06T15:47:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1141665792 bytes 2023-12-06T15:47:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1193383936 bytes 2023-12-06T15:47:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1235224576 bytes 2023-12-06T15:47:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1268680704 bytes 2023-12-06T15:47:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1336882176 bytes 2023-12-06T15:47:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1390818304 bytes 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_23 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:47:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_23 128M 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2023-12-06T15:47:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:47:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-12-06T15:47:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:47:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-12-06T15:47:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:47:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-12-06T15:47:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_23': 2023-12-06T15:47:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 133be3e6dc45 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.133be3e6dc45 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:22 2023 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:22 2023 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:22 2023 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c279a46f-dce9-48d0-be3a-726f061d7181 2023-12-06T15:47:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:47:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:47:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_23 2023-12-06T15:47:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:47:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:47:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=5 2023-12-06T15:47:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:47:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:47:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 46896128 bytes 2023-12-06T15:47:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 104501248 bytes 2023-12-06T15:47:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 105120768 bytes 2023-12-06T15:47:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 171101184 bytes 2023-12-06T15:47:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 186172416 bytes 2023-12-06T15:47:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 224152576 bytes 2023-12-06T15:47:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 314585088 bytes 2023-12-06T15:47:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 365688832 bytes 2023-12-06T15:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 396148736 bytes 2023-12-06T15:47:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 422498304 bytes 2023-12-06T15:47:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 490105856 bytes 2023-12-06T15:47:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 507437056 bytes 2023-12-06T15:47:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 534132736 bytes 2023-12-06T15:47:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 622429184 bytes 2023-12-06T15:47:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 623053824 bytes 2023-12-06T15:47:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 679201792 bytes 2023-12-06T15:47:26.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 736208896 bytes 2023-12-06T15:47:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 765614080 bytes 2023-12-06T15:47:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 797832192 bytes 2023-12-06T15:47:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 831940608 bytes 2023-12-06T15:47:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 844690432 bytes 2023-12-06T15:47:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 917324800 bytes 2023-12-06T15:47:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 919883776 bytes 2023-12-06T15:47:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 975943680 bytes 2023-12-06T15:47:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 993120256 bytes 2023-12-06T15:47:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1046607872 bytes 2023-12-06T15:47:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1062064128 bytes 2023-12-06T15:47:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:47:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:47:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:47:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:47:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:47:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_24 2023-12-06T15:47:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-12-06T15:47:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:47:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:47:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_24 128M 2023-12-06T15:47:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:47:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:47:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:47:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:47:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2023-12-06T15:47:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:47:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-12-06T15:47:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:47:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-12-06T15:47:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:47:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-12-06T15:47:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_24': 2023-12-06T15:47:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:47:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:47:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:47:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 13476f845644 2023-12-06T15:47:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.13476f845644 2023-12-06T15:47:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:47:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:47:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:47:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:47:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:28 2023 2023-12-06T15:47:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:28 2023 2023-12-06T15:47:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:28 2023 2023-12-06T15:47:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:47:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:47:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: a3bf4bbf-7635-41c7-8700-c2fe78a3fbe0 2023-12-06T15:47:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:47:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:47:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_24 2023-12-06T15:47:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:47:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:47:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=9 2023-12-06T15:47:28.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:47:28.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:47:29.232 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-06T15:47:29.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-06T15:47:29.233 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-06T15:47:29.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:47:29.232+0000 7f2bbdac9700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 127671) UID: 1000 2023-12-06T15:47:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 70422528 bytes 2023-12-06T15:47:29.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:47:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 106886144 bytes 2023-12-06T15:47:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 170993664 bytes 2023-12-06T15:47:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 171607040 bytes 2023-12-06T15:47:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 208205824 bytes 2023-12-06T15:47:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 260546560 bytes 2023-12-06T15:47:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 299777024 bytes 2023-12-06T15:47:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 402870272 bytes 2023-12-06T15:47:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 432120832 bytes 2023-12-06T15:47:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 476463104 bytes 2023-12-06T15:47:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 518118400 bytes 2023-12-06T15:47:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 545679360 bytes 2023-12-06T15:47:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 558295040 bytes 2023-12-06T15:47:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 623875072 bytes 2023-12-06T15:47:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 677318656 bytes 2023-12-06T15:47:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 734579712 bytes 2023-12-06T15:47:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 788932608 bytes 2023-12-06T15:47:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 811251712 bytes 2023-12-06T15:47:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 842801152 bytes 2023-12-06T15:47:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 850755584 bytes 2023-12-06T15:47:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 878878720 bytes 2023-12-06T15:47:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 904996864 bytes 2023-12-06T15:47:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 960021504 bytes 2023-12-06T15:47:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 975023104 bytes 2023-12-06T15:47:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1053074432 bytes 2023-12-06T15:47:33.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1143384064 bytes 2023-12-06T15:47:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1163089920 bytes 2023-12-06T15:47:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1233953792 bytes 2023-12-06T15:47:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1246555136 bytes 2023-12-06T15:47:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1274656768 bytes 2023-12-06T15:47:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1342436352 bytes 2023-12-06T15:47:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1368845312 bytes 2023-12-06T15:47:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1392811008 bytes 2023-12-06T15:47:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1432247296 bytes 2023-12-06T15:47:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1511015424 bytes 2023-12-06T15:47:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1543527424 bytes 2023-12-06T15:47:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1585079296 bytes 2023-12-06T15:47:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1658474496 bytes 2023-12-06T15:47:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1659388928 bytes 2023-12-06T15:47:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1682384896 bytes 2023-12-06T15:47:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1717146624 bytes 2023-12-06T15:47:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1752813568 bytes 2023-12-06T15:47:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1792519168 bytes 2023-12-06T15:47:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1850990592 bytes 2023-12-06T15:47:37.233 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-06T15:47:37.234 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:47:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:47:37.234 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-06T15:47:37.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-06T15:47:37.234 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-06T15:47:37.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-06T15:47:37.240 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-12-06T15:47:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1883859968 bytes 2023-12-06T15:47:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1898005504 bytes 2023-12-06T15:47:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1914981376 bytes 2023-12-06T15:47:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1980565504 bytes 2023-12-06T15:47:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:47:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:47:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:47:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:47:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:47:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_25 2023-12-06T15:47:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_25 128M 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:47:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:47:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:47:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:47:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2023-12-06T15:47:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:47:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-12-06T15:47:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:47:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-12-06T15:47:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:47:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-12-06T15:47:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_25': 2023-12-06T15:47:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:47:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:47:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:47:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 13596cf9ebd4 2023-12-06T15:47:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.13596cf9ebd4 2023-12-06T15:47:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:47:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:47:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:47:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:47:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:38 2023 2023-12-06T15:47:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:38 2023 2023-12-06T15:47:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:38 2023 2023-12-06T15:47:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:47:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:47:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: db2fb06f-c4e5-4af2-8764-3691b6d24393 2023-12-06T15:47:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:47:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:47:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_25 2023-12-06T15:47:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:47:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:47:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=8 2023-12-06T15:47:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:47:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:47:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 39363584 bytes 2023-12-06T15:47:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 157086720 bytes 2023-12-06T15:47:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 269014016 bytes 2023-12-06T15:47:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 282459136 bytes 2023-12-06T15:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 306426880 bytes 2023-12-06T15:47:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 338822144 bytes 2023-12-06T15:47:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 355480576 bytes 2023-12-06T15:47:40.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 367579136 bytes 2023-12-06T15:47:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 440130560 bytes 2023-12-06T15:47:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 460637184 bytes 2023-12-06T15:47:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 521474048 bytes 2023-12-06T15:47:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 545801216 bytes 2023-12-06T15:47:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 569915392 bytes 2023-12-06T15:47:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 612020224 bytes 2023-12-06T15:47:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 627446784 bytes 2023-12-06T15:47:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 674608128 bytes 2023-12-06T15:47:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 675241984 bytes 2023-12-06T15:47:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 694210560 bytes 2023-12-06T15:47:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 778131456 bytes 2023-12-06T15:47:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 778767360 bytes 2023-12-06T15:47:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 794880000 bytes 2023-12-06T15:47:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 870403072 bytes 2023-12-06T15:47:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 971720704 bytes 2023-12-06T15:47:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 984861696 bytes 2023-12-06T15:47:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1008393216 bytes 2023-12-06T15:47:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1042350080 bytes 2023-12-06T15:47:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1111066624 bytes 2023-12-06T15:47:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1144419328 bytes 2023-12-06T15:47:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1183904768 bytes 2023-12-06T15:47:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1225373696 bytes 2023-12-06T15:47:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1245444096 bytes 2023-12-06T15:47:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1251075072 bytes 2023-12-06T15:47:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1319838720 bytes 2023-12-06T15:47:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1359544320 bytes 2023-12-06T15:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1406966784 bytes 2023-12-06T15:47:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1424865280 bytes 2023-12-06T15:47:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1486013440 bytes 2023-12-06T15:47:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1524082688 bytes 2023-12-06T15:47:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1564475392 bytes 2023-12-06T15:47:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1654411264 bytes 2023-12-06T15:47:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1709520896 bytes 2023-12-06T15:47:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1728831488 bytes 2023-12-06T15:47:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:47:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:47:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:47:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:47:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:47:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_26 2023-12-06T15:47:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-12-06T15:47:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_26 128M 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:47:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:47:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2023-12-06T15:47:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:47:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-12-06T15:47:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:47:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-12-06T15:47:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:47:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-12-06T15:47:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_26': 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 13659bf364bc 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.13659bf364bc 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:47 2023 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:47 2023 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:47 2023 2023-12-06T15:47:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:47:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:47:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 8f93da9b-50b8-4808-bf76-a4022502be07 2023-12-06T15:47:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:47:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:47:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_26 2023-12-06T15:47:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:47:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:47:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=9 2023-12-06T15:47:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:47:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:47:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 33101824 bytes 2023-12-06T15:47:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 77475840 bytes 2023-12-06T15:47:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 151549952 bytes 2023-12-06T15:47:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 173507584 bytes 2023-12-06T15:47:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 197702656 bytes 2023-12-06T15:47:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 211214336 bytes 2023-12-06T15:47:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 226065408 bytes 2023-12-06T15:47:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 270962688 bytes 2023-12-06T15:47:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 328604672 bytes 2023-12-06T15:47:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 334048256 bytes 2023-12-06T15:47:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 409145344 bytes 2023-12-06T15:47:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 419324928 bytes 2023-12-06T15:47:50.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 447485952 bytes 2023-12-06T15:47:50.323 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3557496046831097) > weight (0.25) 2023-12-06T15:47:50.323 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5905566793459973) > weight (0.25) 2023-12-06T15:47:50.323 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7917772233145344) > weight (0.25) 2023-12-06T15:47:50.323 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4611645295966009) > weight (0.25) 2023-12-06T15:47:50.324 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2023-12-06T15:47:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 482846720 bytes 2023-12-06T15:47:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 522949632 bytes 2023-12-06T15:47:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 551785472 bytes 2023-12-06T15:47:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 591848448 bytes 2023-12-06T15:47:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 592459776 bytes 2023-12-06T15:47:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 637411328 bytes 2023-12-06T15:47:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 660073472 bytes 2023-12-06T15:47:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 688661504 bytes 2023-12-06T15:47:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 732366848 bytes 2023-12-06T15:47:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 743991296 bytes 2023-12-06T15:47:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 763479040 bytes 2023-12-06T15:47:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 799059968 bytes 2023-12-06T15:47:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 836347904 bytes 2023-12-06T15:47:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 900313088 bytes 2023-12-06T15:47:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 965186560 bytes 2023-12-06T15:47:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 966063104 bytes 2023-12-06T15:47:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 978080768 bytes 2023-12-06T15:47:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1025195008 bytes 2023-12-06T15:47:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1025811456 bytes 2023-12-06T15:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1106025472 bytes 2023-12-06T15:47:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1142352896 bytes 2023-12-06T15:47:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1159816192 bytes 2023-12-06T15:47:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1189157888 bytes 2023-12-06T15:47:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1236955136 bytes 2023-12-06T15:47:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1293239296 bytes 2023-12-06T15:47:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1321272320 bytes 2023-12-06T15:47:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1354165248 bytes 2023-12-06T15:47:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1369464832 bytes 2023-12-06T15:47:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1420258304 bytes 2023-12-06T15:47:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1437679616 bytes 2023-12-06T15:47:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1530660864 bytes 2023-12-06T15:47:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1558340608 bytes 2023-12-06T15:47:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1601598464 bytes 2023-12-06T15:47:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1656835072 bytes 2023-12-06T15:47:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1710114816 bytes 2023-12-06T15:47:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:47:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:47:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:47:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:47:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:47:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_27 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_27 128M 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:47:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:47:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:47:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:47:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:47:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2023-12-06T15:47:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-12-06T15:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-12-06T15:47:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:47:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-12-06T15:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_27': 2023-12-06T15:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 13713c76cad4 2023-12-06T15:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.13713c76cad4 2023-12-06T15:47:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:57 2023 2023-12-06T15:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:57 2023 2023-12-06T15:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:57 2023 2023-12-06T15:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c0035e14-7c4a-401d-a289-cee1fa7ce091 2023-12-06T15:47:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:47:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:47:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_27 2023-12-06T15:47:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:47:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:47:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:47:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:47:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=7 2023-12-06T15:47:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:47:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:47:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 54933504 bytes 2023-12-06T15:47:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 88350720 bytes 2023-12-06T15:47:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 126602240 bytes 2023-12-06T15:47:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 163524608 bytes 2023-12-06T15:47:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 216328192 bytes 2023-12-06T15:47:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 222260224 bytes 2023-12-06T15:47:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 260797440 bytes 2023-12-06T15:47:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 286192640 bytes 2023-12-06T15:47:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 336814080 bytes 2023-12-06T15:47:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 359042048 bytes 2023-12-06T15:48:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 438714368 bytes 2023-12-06T15:48:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 513642496 bytes 2023-12-06T15:48:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 557755392 bytes 2023-12-06T15:48:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 605911040 bytes 2023-12-06T15:48:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 616909824 bytes 2023-12-06T15:48:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 639814656 bytes 2023-12-06T15:48:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 693065728 bytes 2023-12-06T15:48:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 746284032 bytes 2023-12-06T15:48:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 774121472 bytes 2023-12-06T15:48:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 777402368 bytes 2023-12-06T15:48:01.679 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:48:01.678+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e9123ee820 2023-12-06T15:48:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 805648384 bytes 2023-12-06T15:48:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 833517568 bytes 2023-12-06T15:48:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 871694336 bytes 2023-12-06T15:48:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 918270976 bytes 2023-12-06T15:48:02.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1021868032 bytes 2023-12-06T15:48:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1099925504 bytes 2023-12-06T15:48:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1130198016 bytes 2023-12-06T15:48:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1197313024 bytes 2023-12-06T15:48:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1252292608 bytes 2023-12-06T15:48:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1304025088 bytes 2023-12-06T15:48:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1340495872 bytes 2023-12-06T15:48:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1382252544 bytes 2023-12-06T15:48:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1393453056 bytes 2023-12-06T15:48:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1428392960 bytes 2023-12-06T15:48:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1462436864 bytes 2023-12-06T15:48:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1487472640 bytes 2023-12-06T15:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1527441408 bytes 2023-12-06T15:48:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:48:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:48:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:48:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:48:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:48:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_28 2023-12-06T15:48:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-12-06T15:48:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_28 128M 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:48:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2023-12-06T15:48:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:48:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-12-06T15:48:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:48:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:48:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-12-06T15:48:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:48:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_28': 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 137da5208307 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.137da5208307 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:48:05 2023 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:48:05 2023 2023-12-06T15:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:48:05 2023 2023-12-06T15:48:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:48:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:48:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 45a35e04-1f3a-4f6c-a378-30b333fab016 2023-12-06T15:48:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:48:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:48:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_28 2023-12-06T15:48:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:48:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:48:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=7 2023-12-06T15:48:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:48:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:48:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 17784832 bytes 2023-12-06T15:48:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 41947136 bytes 2023-12-06T15:48:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 66203648 bytes 2023-12-06T15:48:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 133156864 bytes 2023-12-06T15:48:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 138397696 bytes 2023-12-06T15:48:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 174835712 bytes 2023-12-06T15:48:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 196971520 bytes 2023-12-06T15:48:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 278021120 bytes 2023-12-06T15:48:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 334148608 bytes 2023-12-06T15:48:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 365379584 bytes 2023-12-06T15:48:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 401548288 bytes 2023-12-06T15:48:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 421656576 bytes 2023-12-06T15:48:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 477250560 bytes 2023-12-06T15:48:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 515258368 bytes 2023-12-06T15:48:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 546610176 bytes 2023-12-06T15:48:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 575783936 bytes 2023-12-06T15:48:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 601164800 bytes 2023-12-06T15:48:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 601789440 bytes 2023-12-06T15:48:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 631355392 bytes 2023-12-06T15:48:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 636617728 bytes 2023-12-06T15:48:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 651379712 bytes 2023-12-06T15:48:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 696271872 bytes 2023-12-06T15:48:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 743853056 bytes 2023-12-06T15:48:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 795844608 bytes 2023-12-06T15:48:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 808599552 bytes 2023-12-06T15:48:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 854178816 bytes 2023-12-06T15:48:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 912540672 bytes 2023-12-06T15:48:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1005637632 bytes 2023-12-06T15:48:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1034842112 bytes 2023-12-06T15:48:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1089754112 bytes 2023-12-06T15:48:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1126626304 bytes 2023-12-06T15:48:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1216740352 bytes 2023-12-06T15:48:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1225341952 bytes 2023-12-06T15:48:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1264777216 bytes 2023-12-06T15:48:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1295465472 bytes 2023-12-06T15:48:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1324866560 bytes 2023-12-06T15:48:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1377269760 bytes 2023-12-06T15:48:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1456910336 bytes 2023-12-06T15:48:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:48:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:48:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:48:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_29 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_29 128M 2023-12-06T15:48:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:48:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:48:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:48:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:48:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2023-12-06T15:48:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:48:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-12-06T15:48:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:48:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:48:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-12-06T15:48:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:48:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-12-06T15:48:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_29': 2023-12-06T15:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 1389e36c4bc7 2023-12-06T15:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.1389e36c4bc7 2023-12-06T15:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:48:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:48:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:48:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:48:13 2023 2023-12-06T15:48:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:48:13 2023 2023-12-06T15:48:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:48:13 2023 2023-12-06T15:48:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:48:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:48:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 0031e62d-baea-4f3c-837b-04d500983bed 2023-12-06T15:48:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:48:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:48:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_29 2023-12-06T15:48:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:48:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:48:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=9 2023-12-06T15:48:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:48:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:48:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 11174912 bytes 2023-12-06T15:48:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 36373504 bytes 2023-12-06T15:48:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 102144000 bytes 2023-12-06T15:48:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 102764544 bytes 2023-12-06T15:48:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 121767936 bytes 2023-12-06T15:48:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 201195520 bytes 2023-12-06T15:48:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 252649472 bytes 2023-12-06T15:48:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 308319232 bytes 2023-12-06T15:48:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 332646400 bytes 2023-12-06T15:48:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 359407616 bytes 2023-12-06T15:48:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 451130368 bytes 2023-12-06T15:48:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 537994240 bytes 2023-12-06T15:48:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 554679296 bytes 2023-12-06T15:48:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 568906752 bytes 2023-12-06T15:48:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 649687040 bytes 2023-12-06T15:48:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 704624640 bytes 2023-12-06T15:48:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 817184768 bytes 2023-12-06T15:48:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 898169856 bytes 2023-12-06T15:48:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 905205760 bytes 2023-12-06T15:48:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 923235328 bytes 2023-12-06T15:48:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 958712832 bytes 2023-12-06T15:48:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 984281088 bytes 2023-12-06T15:48:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1021079552 bytes 2023-12-06T15:48:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1085390848 bytes 2023-12-06T15:48:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1159068672 bytes 2023-12-06T15:48:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1234302976 bytes 2023-12-06T15:48:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1315480576 bytes 2023-12-06T15:48:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1357959168 bytes 2023-12-06T15:48:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1385092096 bytes 2023-12-06T15:48:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1433123840 bytes 2023-12-06T15:48:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1497943040 bytes 2023-12-06T15:48:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1545259008 bytes 2023-12-06T15:48:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1598607360 bytes 2023-12-06T15:48:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1618725888 bytes 2023-12-06T15:48:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1642016768 bytes 2023-12-06T15:48:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1672307712 bytes 2023-12-06T15:48:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1735520256 bytes 2023-12-06T15:48:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1799224320 bytes 2023-12-06T15:48:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1804986368 bytes 2023-12-06T15:48:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1923287040 bytes 2023-12-06T15:48:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1954108416 bytes 2023-12-06T15:48:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1954688000 bytes 2023-12-06T15:48:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 2005599232 bytes 2023-12-06T15:48:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 2055314432 bytes 2023-12-06T15:48:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 2076741632 bytes 2023-12-06T15:48:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:48:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:48:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_30 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:48:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_30 128M 2023-12-06T15:48:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:48:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:48:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:48:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:48:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2023-12-06T15:48:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:48:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-12-06T15:48:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:48:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:48:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-12-06T15:48:24.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:48:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-12-06T15:48:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_30': 2023-12-06T15:48:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:48:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:48:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:48:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 1395ac20caf4 2023-12-06T15:48:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.1395ac20caf4 2023-12-06T15:48:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:48:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:48:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:48:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:48:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:48:23 2023 2023-12-06T15:48:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:48:23 2023 2023-12-06T15:48:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:48:23 2023 2023-12-06T15:48:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:48:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:48:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 393bd03f-74d9-4863-9753-476b06c160ad 2023-12-06T15:48:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:48:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:48:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_30 2023-12-06T15:48:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:48:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:48:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=12 2023-12-06T15:48:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:48:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:48:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 61391872 bytes 2023-12-06T15:48:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 72604672 bytes 2023-12-06T15:48:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 82534400 bytes 2023-12-06T15:48:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 83142656 bytes 2023-12-06T15:48:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 115934208 bytes 2023-12-06T15:48:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 152091648 bytes 2023-12-06T15:48:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 235845632 bytes 2023-12-06T15:48:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 255459328 bytes 2023-12-06T15:48:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 335037440 bytes 2023-12-06T15:48:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 363509760 bytes 2023-12-06T15:48:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 364109824 bytes 2023-12-06T15:48:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 405356544 bytes 2023-12-06T15:48:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 461308928 bytes 2023-12-06T15:48:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 506473472 bytes 2023-12-06T15:48:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 521565184 bytes 2023-12-06T15:48:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 559949824 bytes 2023-12-06T15:48:27.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 597453824 bytes 2023-12-06T15:48:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 660224000 bytes 2023-12-06T15:48:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 679811072 bytes 2023-12-06T15:48:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 721737728 bytes 2023-12-06T15:48:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 779608064 bytes 2023-12-06T15:48:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 803077120 bytes 2023-12-06T15:48:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 855059456 bytes 2023-12-06T15:48:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 924630016 bytes 2023-12-06T15:48:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 953854976 bytes 2023-12-06T15:48:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1015176192 bytes 2023-12-06T15:48:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1036571648 bytes 2023-12-06T15:48:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1134065664 bytes 2023-12-06T15:48:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1186358272 bytes 2023-12-06T15:48:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1225184256 bytes 2023-12-06T15:48:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1278407680 bytes 2023-12-06T15:48:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1303206912 bytes 2023-12-06T15:48:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1323464704 bytes 2023-12-06T15:48:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1366250496 bytes 2023-12-06T15:48:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1398069248 bytes 2023-12-06T15:48:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1411457024 bytes 2023-12-06T15:48:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1440417792 bytes 2023-12-06T15:48:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1481861120 bytes 2023-12-06T15:48:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1495942144 bytes 2023-12-06T15:48:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1498047488 bytes 2023-12-06T15:48:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1561065472 bytes 2023-12-06T15:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1638688768 bytes 2023-12-06T15:48:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1669446656 bytes 2023-12-06T15:48:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1739308032 bytes 2023-12-06T15:48:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1771407360 bytes 2023-12-06T15:48:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1814096896 bytes 2023-12-06T15:48:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1814704128 bytes 2023-12-06T15:48:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1842430976 bytes 2023-12-06T15:48:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1962797056 bytes 2023-12-06T15:48:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1999883264 bytes 2023-12-06T15:48:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2039329792 bytes 2023-12-06T15:48:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2039933952 bytes 2023-12-06T15:48:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2045988864 bytes 2023-12-06T15:48:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2074779648 bytes 2023-12-06T15:48:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2098491392 bytes 2023-12-06T15:48:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2124041216 bytes 2023-12-06T15:48:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2185392128 bytes 2023-12-06T15:48:34.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2226360320 bytes 2023-12-06T15:48:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2240748544 bytes 2023-12-06T15:48:35.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2378884096 bytes 2023-12-06T15:48:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2402667520 bytes 2023-12-06T15:48:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2464206848 bytes 2023-12-06T15:48:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2494255104 bytes 2023-12-06T15:48:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2565424128 bytes 2023-12-06T15:48:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:48:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:48:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:48:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:48:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:48:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_31 2023-12-06T15:48:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-12-06T15:48:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_31 128M 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2023-12-06T15:48:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:48:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-12-06T15:48:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:48:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:48:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-12-06T15:48:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-12-06T15:48:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_31': 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 13a13b3daa5f 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.13a13b3daa5f 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:48:36 2023 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:48:36 2023 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:48:36 2023 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:48:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:48:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c5f2021b-abb5-4a19-80cd-cedca15b14e1 2023-12-06T15:48:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_31 2023-12-06T15:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:48:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:48:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=8 2023-12-06T15:48:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:48:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:48:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 24255488 bytes 2023-12-06T15:48:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 42274816 bytes 2023-12-06T15:48:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 93060096 bytes 2023-12-06T15:48:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 122345472 bytes 2023-12-06T15:48:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 146155520 bytes 2023-12-06T15:48:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 169879552 bytes 2023-12-06T15:48:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 208851968 bytes 2023-12-06T15:48:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 216478720 bytes 2023-12-06T15:48:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 259629056 bytes 2023-12-06T15:48:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 291294208 bytes 2023-12-06T15:48:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 329575424 bytes 2023-12-06T15:48:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 358904832 bytes 2023-12-06T15:48:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 419852288 bytes 2023-12-06T15:48:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 462901248 bytes 2023-12-06T15:48:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 486925312 bytes 2023-12-06T15:48:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 515062784 bytes 2023-12-06T15:48:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 533650432 bytes 2023-12-06T15:48:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 562822144 bytes 2023-12-06T15:48:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 628670464 bytes 2023-12-06T15:48:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 672411648 bytes 2023-12-06T15:48:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 742455296 bytes 2023-12-06T15:48:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 791802880 bytes 2023-12-06T15:48:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 833292288 bytes 2023-12-06T15:48:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 863127552 bytes 2023-12-06T15:48:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 882534400 bytes 2023-12-06T15:48:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 919323648 bytes 2023-12-06T15:48:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 970146816 bytes 2023-12-06T15:48:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1004442624 bytes 2023-12-06T15:48:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1047336960 bytes 2023-12-06T15:48:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1068336128 bytes 2023-12-06T15:48:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1117030400 bytes 2023-12-06T15:48:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1161345024 bytes 2023-12-06T15:48:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1178145792 bytes 2023-12-06T15:48:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1265126400 bytes 2023-12-06T15:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1332118528 bytes 2023-12-06T15:48:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1380836352 bytes 2023-12-06T15:48:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1405977600 bytes 2023-12-06T15:48:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1448350720 bytes 2023-12-06T15:48:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1509943296 bytes 2023-12-06T15:48:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1600150528 bytes 2023-12-06T15:48:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1625236480 bytes 2023-12-06T15:48:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1628672000 bytes 2023-12-06T15:48:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1679201280 bytes 2023-12-06T15:48:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:48:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:48:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:48:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:48:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_32 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_32 128M 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:48:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:48:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:48:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:48:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2023-12-06T15:48:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:48:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-12-06T15:48:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:48:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:48:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-12-06T15:48:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:48:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-12-06T15:48:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_32': 2023-12-06T15:48:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:48:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:48:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:48:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 13ad7018a798 2023-12-06T15:48:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.13ad7018a798 2023-12-06T15:48:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:48:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:48:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:48:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:48:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:48:45 2023 2023-12-06T15:48:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:48:45 2023 2023-12-06T15:48:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:48:45 2023 2023-12-06T15:48:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:48:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:48:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 994706d1-52be-4379-8f1d-c660ef60435c 2023-12-06T15:48:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:48:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:48:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_32 2023-12-06T15:48:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:48:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:48:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=13 2023-12-06T15:48:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:48:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:48:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 43360256 bytes 2023-12-06T15:48:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 84322304 bytes 2023-12-06T15:48:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 115614720 bytes 2023-12-06T15:48:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 126699520 bytes 2023-12-06T15:48:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 201894912 bytes 2023-12-06T15:48:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 303762432 bytes 2023-12-06T15:48:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 311560192 bytes 2023-12-06T15:48:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 342434816 bytes 2023-12-06T15:48:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 367339520 bytes 2023-12-06T15:48:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 380756992 bytes 2023-12-06T15:48:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 402279424 bytes 2023-12-06T15:48:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 538091520 bytes 2023-12-06T15:48:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 586002432 bytes 2023-12-06T15:48:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 626983936 bytes 2023-12-06T15:48:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 687087616 bytes 2023-12-06T15:48:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 774416384 bytes 2023-12-06T15:48:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 823093248 bytes 2023-12-06T15:48:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 869743616 bytes 2023-12-06T15:48:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 880513024 bytes 2023-12-06T15:48:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 928375808 bytes 2023-12-06T15:48:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 981823488 bytes 2023-12-06T15:48:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1005211648 bytes 2023-12-06T15:48:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1030606848 bytes 2023-12-06T15:48:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1061150720 bytes 2023-12-06T15:48:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1132475392 bytes 2023-12-06T15:48:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1205497856 bytes 2023-12-06T15:48:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1266019328 bytes 2023-12-06T15:48:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1300205568 bytes 2023-12-06T15:48:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1365623808 bytes 2023-12-06T15:48:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1390399488 bytes 2023-12-06T15:48:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1404468224 bytes 2023-12-06T15:48:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1460474880 bytes 2023-12-06T15:48:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1520442368 bytes 2023-12-06T15:48:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1540713472 bytes 2023-12-06T15:48:53.324 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7191781284468164) > weight (0.25) 2023-12-06T15:48:53.325 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9514746844998573) > weight (0.25) 2023-12-06T15:48:53.325 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3821167202481116) > weight (0.25) 2023-12-06T15:48:53.325 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7803281342957472) > weight (0.25) 2023-12-06T15:48:53.325 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-12-06T15:48:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1585304576 bytes 2023-12-06T15:48:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1612631040 bytes 2023-12-06T15:48:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1679807488 bytes 2023-12-06T15:48:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1684727808 bytes 2023-12-06T15:48:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1685327872 bytes 2023-12-06T15:48:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1690335232 bytes 2023-12-06T15:48:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1715083264 bytes 2023-12-06T15:48:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1715697664 bytes 2023-12-06T15:48:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1747906560 bytes 2023-12-06T15:48:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1808351232 bytes 2023-12-06T15:48:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1825974272 bytes 2023-12-06T15:48:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1853322240 bytes 2023-12-06T15:48:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1877154816 bytes 2023-12-06T15:48:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1956303872 bytes 2023-12-06T15:48:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2003296256 bytes 2023-12-06T15:48:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2005181440 bytes 2023-12-06T15:48:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2033486848 bytes 2023-12-06T15:48:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2072784896 bytes 2023-12-06T15:48:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2073389056 bytes 2023-12-06T15:48:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2093519872 bytes 2023-12-06T15:48:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2137362432 bytes 2023-12-06T15:48:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2181995520 bytes 2023-12-06T15:48:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2200156160 bytes 2023-12-06T15:48:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2225866752 bytes 2023-12-06T15:48:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2250542080 bytes 2023-12-06T15:48:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2294223872 bytes 2023-12-06T15:48:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2328067072 bytes 2023-12-06T15:48:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2372282368 bytes 2023-12-06T15:48:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2431388672 bytes 2023-12-06T15:48:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2487365632 bytes 2023-12-06T15:48:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 2541665280 bytes 2023-12-06T15:48:58.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6600 IOs, 2581338112 bytes 2023-12-06T15:48:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6700 IOs, 2623960064 bytes 2023-12-06T15:48:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6800 IOs, 2701352960 bytes 2023-12-06T15:48:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_33 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:48:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_33 128M 2023-12-06T15:48:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:48:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:48:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2023-12-06T15:48:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:48:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-12-06T15:48:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:48:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:48:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:48:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:48:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-12-06T15:49:00.241 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-06T15:49:00.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-06T15:49:00.242 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-06T15:49:00.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:49:00.241+0000 7f33b2c24700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 132022) UID: 1000 2023-12-06T15:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:49:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-12-06T15:49:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_33': 2023-12-06T15:49:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:49:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:49:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:49:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 13b91e1827d2 2023-12-06T15:49:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.13b91e1827d2 2023-12-06T15:49:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:49:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:49:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:49:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:49:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:48:59 2023 2023-12-06T15:49:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:48:59 2023 2023-12-06T15:49:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:48:59 2023 2023-12-06T15:49:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:49:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:49:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 3293bbff-63fc-47b0-b7af-aa270d848ce2 2023-12-06T15:49:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:49:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:49:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_33 2023-12-06T15:49:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:49:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:49:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=6 2023-12-06T15:49:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:49:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:49:00.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:49:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 36951040 bytes 2023-12-06T15:49:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 71156736 bytes 2023-12-06T15:49:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 113030144 bytes 2023-12-06T15:49:01.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 139655168 bytes 2023-12-06T15:49:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 183521280 bytes 2023-12-06T15:49:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 223302656 bytes 2023-12-06T15:49:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 261335040 bytes 2023-12-06T15:49:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 298195968 bytes 2023-12-06T15:49:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 327463936 bytes 2023-12-06T15:49:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 382312448 bytes 2023-12-06T15:49:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 423209984 bytes 2023-12-06T15:49:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 450491392 bytes 2023-12-06T15:49:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 509798400 bytes 2023-12-06T15:49:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 525868032 bytes 2023-12-06T15:49:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 568392704 bytes 2023-12-06T15:49:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 586451968 bytes 2023-12-06T15:49:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 631904256 bytes 2023-12-06T15:49:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 663931904 bytes 2023-12-06T15:49:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 717068288 bytes 2023-12-06T15:49:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 737561600 bytes 2023-12-06T15:49:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 791938048 bytes 2023-12-06T15:49:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 820375552 bytes 2023-12-06T15:49:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 820977664 bytes 2023-12-06T15:49:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 842546176 bytes 2023-12-06T15:49:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 866582528 bytes 2023-12-06T15:49:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 887036928 bytes 2023-12-06T15:49:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 903803904 bytes 2023-12-06T15:49:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 929686528 bytes 2023-12-06T15:49:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1009304576 bytes 2023-12-06T15:49:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1070840832 bytes 2023-12-06T15:49:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1083668480 bytes 2023-12-06T15:49:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1144801280 bytes 2023-12-06T15:49:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1186792448 bytes 2023-12-06T15:49:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:49:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:49:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:49:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:49:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:49:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_34 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_34 128M 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:49:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:49:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:49:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:49:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2023-12-06T15:49:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:49:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-12-06T15:49:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:49:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:49:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-12-06T15:49:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:49:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-12-06T15:49:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_34': 2023-12-06T15:49:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:49:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:49:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:49:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 13c57327579b 2023-12-06T15:49:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.13c57327579b 2023-12-06T15:49:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:49:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:49:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:49:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:49:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:49:06 2023 2023-12-06T15:49:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:49:06 2023 2023-12-06T15:49:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:49:06 2023 2023-12-06T15:49:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:49:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:49:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: dc0720e9-3db1-4118-8a6c-3ea4239b21d7 2023-12-06T15:49:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:49:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:49:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_34 2023-12-06T15:49:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:49:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:49:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=11 2023-12-06T15:49:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:49:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:49:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 51469312 bytes 2023-12-06T15:49:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 89350144 bytes 2023-12-06T15:49:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 108521472 bytes 2023-12-06T15:49:08.242 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-06T15:49:08.243 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:49:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:49:08.243 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-06T15:49:08.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-06T15:49:08.244 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-06T15:49:08.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-06T15:49:08.246 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2023-12-06T15:49:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 129299456 bytes 2023-12-06T15:49:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 171904000 bytes 2023-12-06T15:49:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 257381376 bytes 2023-12-06T15:49:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 293229568 bytes 2023-12-06T15:49:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 365621248 bytes 2023-12-06T15:49:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 393333760 bytes 2023-12-06T15:49:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 434064384 bytes 2023-12-06T15:49:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 474182656 bytes 2023-12-06T15:49:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 487517184 bytes 2023-12-06T15:49:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 554993664 bytes 2023-12-06T15:49:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 592490496 bytes 2023-12-06T15:49:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 640513024 bytes 2023-12-06T15:49:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 660178944 bytes 2023-12-06T15:49:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 717803520 bytes 2023-12-06T15:49:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 722427904 bytes 2023-12-06T15:49:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 723032064 bytes 2023-12-06T15:49:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 806719488 bytes 2023-12-06T15:49:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 853186560 bytes 2023-12-06T15:49:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 903337984 bytes 2023-12-06T15:49:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 913576960 bytes 2023-12-06T15:49:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 983410688 bytes 2023-12-06T15:49:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1046923264 bytes 2023-12-06T15:49:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1084800000 bytes 2023-12-06T15:49:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1118774272 bytes 2023-12-06T15:49:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1154018304 bytes 2023-12-06T15:49:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1237685248 bytes 2023-12-06T15:49:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1262013440 bytes 2023-12-06T15:49:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1294190592 bytes 2023-12-06T15:49:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1377597440 bytes 2023-12-06T15:49:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1408081920 bytes 2023-12-06T15:49:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1442818048 bytes 2023-12-06T15:49:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1456814080 bytes 2023-12-06T15:49:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1468505088 bytes 2023-12-06T15:49:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1514915840 bytes 2023-12-06T15:49:14.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1538568192 bytes 2023-12-06T15:49:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1588887552 bytes 2023-12-06T15:49:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1672094720 bytes 2023-12-06T15:49:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1691927552 bytes 2023-12-06T15:49:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1751822336 bytes 2023-12-06T15:49:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1771555840 bytes 2023-12-06T15:49:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1775473664 bytes 2023-12-06T15:49:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1826172928 bytes 2023-12-06T15:49:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1847587840 bytes 2023-12-06T15:49:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1916865536 bytes 2023-12-06T15:49:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1925214208 bytes 2023-12-06T15:49:16.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1925838848 bytes 2023-12-06T15:49:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1952246784 bytes 2023-12-06T15:49:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 1979533312 bytes 2023-12-06T15:49:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2115653632 bytes 2023-12-06T15:49:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2150250496 bytes 2023-12-06T15:49:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2198711296 bytes 2023-12-06T15:49:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2214168576 bytes 2023-12-06T15:49:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2266803200 bytes 2023-12-06T15:49:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:49:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:49:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:49:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:49:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:49:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_35 2023-12-06T15:49:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-12-06T15:49:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_35 128M 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:49:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:49:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2023-12-06T15:49:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:49:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-12-06T15:49:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:49:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:49:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-12-06T15:49:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:49:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-12-06T15:49:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_35': 2023-12-06T15:49:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:49:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:49:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:49:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 13d525f7566a 2023-12-06T15:49:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.13d525f7566a 2023-12-06T15:49:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:49:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:49:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:49:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:49:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:49:18 2023 2023-12-06T15:49:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:49:18 2023 2023-12-06T15:49:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:49:18 2023 2023-12-06T15:49:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:49:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:49:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 12508ac3-20ec-4ba0-98c2-40a26628f314 2023-12-06T15:49:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:49:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:49:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_35 2023-12-06T15:49:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:49:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:49:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=5 2023-12-06T15:49:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:49:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:49:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 51167232 bytes 2023-12-06T15:49:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 54117376 bytes 2023-12-06T15:49:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 61756416 bytes 2023-12-06T15:49:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 73785344 bytes 2023-12-06T15:49:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 103884800 bytes 2023-12-06T15:49:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 131169280 bytes 2023-12-06T15:49:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 177940480 bytes 2023-12-06T15:49:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 212904960 bytes 2023-12-06T15:49:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 233342976 bytes 2023-12-06T15:49:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 251028480 bytes 2023-12-06T15:49:21.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 324617216 bytes 2023-12-06T15:49:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 369181696 bytes 2023-12-06T15:49:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 419712000 bytes 2023-12-06T15:49:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 452284416 bytes 2023-12-06T15:49:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 534193152 bytes 2023-12-06T15:49:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 549616640 bytes 2023-12-06T15:49:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 585185280 bytes 2023-12-06T15:49:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 647054336 bytes 2023-12-06T15:49:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 677288960 bytes 2023-12-06T15:49:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 720445440 bytes 2023-12-06T15:49:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 758536192 bytes 2023-12-06T15:49:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 780460032 bytes 2023-12-06T15:49:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 835076096 bytes 2023-12-06T15:49:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 851806208 bytes 2023-12-06T15:49:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 860242944 bytes 2023-12-06T15:49:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 899957760 bytes 2023-12-06T15:49:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 922476544 bytes 2023-12-06T15:49:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:49:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:49:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:49:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:49:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:49:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_36 2023-12-06T15:49:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-12-06T15:49:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:49:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:49:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:49:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_36 128M 2023-12-06T15:49:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:49:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:49:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:49:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:49:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2023-12-06T15:49:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:49:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-12-06T15:49:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:49:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:49:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-12-06T15:49:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:49:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-12-06T15:49:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_36': 2023-12-06T15:49:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:49:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:49:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:49:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 13e12bb610a7 2023-12-06T15:49:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.13e12bb610a7 2023-12-06T15:49:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:49:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:49:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:49:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:49:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:49:24 2023 2023-12-06T15:49:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:49:24 2023 2023-12-06T15:49:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:49:24 2023 2023-12-06T15:49:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:49:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:49:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 4ef4b927-2e3c-4c53-87cf-79905d0ea25b 2023-12-06T15:49:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:49:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:49:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_36 2023-12-06T15:49:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:49:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:49:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=12 2023-12-06T15:49:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:49:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:49:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 41570304 bytes 2023-12-06T15:49:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 91465728 bytes 2023-12-06T15:49:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 110435328 bytes 2023-12-06T15:49:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 111057920 bytes 2023-12-06T15:49:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 145732608 bytes 2023-12-06T15:49:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 175813632 bytes 2023-12-06T15:49:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 235444224 bytes 2023-12-06T15:49:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 259104768 bytes 2023-12-06T15:49:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 295239680 bytes 2023-12-06T15:49:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 304229376 bytes 2023-12-06T15:49:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 338458624 bytes 2023-12-06T15:49:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 405243904 bytes 2023-12-06T15:49:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 432435200 bytes 2023-12-06T15:49:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 487358464 bytes 2023-12-06T15:49:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 501761024 bytes 2023-12-06T15:49:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 536370176 bytes 2023-12-06T15:49:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 595251200 bytes 2023-12-06T15:49:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 599874560 bytes 2023-12-06T15:49:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 636833792 bytes 2023-12-06T15:49:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 711880704 bytes 2023-12-06T15:49:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 783314944 bytes 2023-12-06T15:49:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 812574720 bytes 2023-12-06T15:49:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 849837056 bytes 2023-12-06T15:49:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 876512256 bytes 2023-12-06T15:49:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 878698496 bytes 2023-12-06T15:49:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 903166976 bytes 2023-12-06T15:49:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 937800704 bytes 2023-12-06T15:49:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1030779904 bytes 2023-12-06T15:49:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1071724544 bytes 2023-12-06T15:49:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1112179712 bytes 2023-12-06T15:49:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1166093312 bytes 2023-12-06T15:49:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1221493760 bytes 2023-12-06T15:49:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1262520320 bytes 2023-12-06T15:49:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1284939776 bytes 2023-12-06T15:49:32.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1345890304 bytes 2023-12-06T15:49:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1355247616 bytes 2023-12-06T15:49:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1397117952 bytes 2023-12-06T15:49:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1448261632 bytes 2023-12-06T15:49:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1501234176 bytes 2023-12-06T15:49:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1571990528 bytes 2023-12-06T15:49:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1618755584 bytes 2023-12-06T15:49:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1684585472 bytes 2023-12-06T15:49:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1735108608 bytes 2023-12-06T15:49:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1763953664 bytes 2023-12-06T15:49:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1777670144 bytes 2023-12-06T15:49:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1832962048 bytes 2023-12-06T15:49:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1848144896 bytes 2023-12-06T15:49:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1863245824 bytes 2023-12-06T15:49:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1942010880 bytes 2023-12-06T15:49:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1976144896 bytes 2023-12-06T15:49:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2016193536 bytes 2023-12-06T15:49:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2041731072 bytes 2023-12-06T15:49:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2042363904 bytes 2023-12-06T15:49:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2060924928 bytes 2023-12-06T15:49:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2137837568 bytes 2023-12-06T15:49:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2173540352 bytes 2023-12-06T15:49:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2222527488 bytes 2023-12-06T15:49:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2317309952 bytes 2023-12-06T15:49:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2344387584 bytes 2023-12-06T15:49:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2420024320 bytes 2023-12-06T15:49:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2469143552 bytes 2023-12-06T15:49:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2493954048 bytes 2023-12-06T15:49:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:49:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:49:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:49:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:49:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:49:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_37 2023-12-06T15:49:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-12-06T15:49:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_37 128M 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2023-12-06T15:49:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:49:37.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-12-06T15:49:37.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:37.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:37.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:49:37.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:49:37.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-12-06T15:49:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:49:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_37': 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 13eda3ac2fc6 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.13eda3ac2fc6 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:49:37 2023 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:49:37 2023 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:49:37 2023 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:49:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:49:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 9a78f30b-f892-4362-84e4-ebdc27892076 2023-12-06T15:49:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:49:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:49:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_37 2023-12-06T15:49:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:49:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:49:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=13 2023-12-06T15:49:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:49:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:49:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 39649280 bytes 2023-12-06T15:49:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 72218624 bytes 2023-12-06T15:49:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 120169472 bytes 2023-12-06T15:49:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 174954496 bytes 2023-12-06T15:49:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 215335936 bytes 2023-12-06T15:49:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 291965952 bytes 2023-12-06T15:49:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 348732416 bytes 2023-12-06T15:49:40.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 384855040 bytes 2023-12-06T15:49:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 410167296 bytes 2023-12-06T15:49:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 462863360 bytes 2023-12-06T15:49:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 477495296 bytes 2023-12-06T15:49:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 493170688 bytes 2023-12-06T15:49:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 520324096 bytes 2023-12-06T15:49:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 573731840 bytes 2023-12-06T15:49:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 608807936 bytes 2023-12-06T15:49:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 626045952 bytes 2023-12-06T15:49:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 673744896 bytes 2023-12-06T15:49:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 706644992 bytes 2023-12-06T15:49:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 734118912 bytes 2023-12-06T15:49:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 773790720 bytes 2023-12-06T15:49:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 844271616 bytes 2023-12-06T15:49:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 910494720 bytes 2023-12-06T15:49:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 958936064 bytes 2023-12-06T15:49:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1030757376 bytes 2023-12-06T15:49:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1091173376 bytes 2023-12-06T15:49:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1166097408 bytes 2023-12-06T15:49:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1166706688 bytes 2023-12-06T15:49:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1259126784 bytes 2023-12-06T15:49:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1330546688 bytes 2023-12-06T15:49:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1335276544 bytes 2023-12-06T15:49:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1409560576 bytes 2023-12-06T15:49:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1463221248 bytes 2023-12-06T15:49:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1468136448 bytes 2023-12-06T15:49:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1503790080 bytes 2023-12-06T15:49:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1540372480 bytes 2023-12-06T15:49:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1578391552 bytes 2023-12-06T15:49:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1621472256 bytes 2023-12-06T15:49:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1642156032 bytes 2023-12-06T15:49:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1671656448 bytes 2023-12-06T15:49:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1679758336 bytes 2023-12-06T15:49:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1696201728 bytes 2023-12-06T15:49:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1696812032 bytes 2023-12-06T15:49:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1716788224 bytes 2023-12-06T15:49:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1730067456 bytes 2023-12-06T15:49:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1775150080 bytes 2023-12-06T15:49:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1852116992 bytes 2023-12-06T15:49:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1861907456 bytes 2023-12-06T15:49:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1935206400 bytes 2023-12-06T15:49:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1995123712 bytes 2023-12-06T15:49:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2054648832 bytes 2023-12-06T15:49:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2076639232 bytes 2023-12-06T15:49:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2113292288 bytes 2023-12-06T15:49:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2146965504 bytes 2023-12-06T15:49:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2198971392 bytes 2023-12-06T15:49:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2227856384 bytes 2023-12-06T15:49:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2273723392 bytes 2023-12-06T15:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2338955264 bytes 2023-12-06T15:49:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2399864832 bytes 2023-12-06T15:49:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2418904064 bytes 2023-12-06T15:49:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2459867136 bytes 2023-12-06T15:49:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2519070720 bytes 2023-12-06T15:49:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2555250688 bytes 2023-12-06T15:49:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2555870208 bytes 2023-12-06T15:49:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2597489664 bytes 2023-12-06T15:49:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 2634977280 bytes 2023-12-06T15:49:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:49:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:49:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_38 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_38 128M 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:49:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:49:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:49:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:49:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2023-12-06T15:49:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:49:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-12-06T15:49:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:49:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:49:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-12-06T15:49:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:49:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-12-06T15:49:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_38': 2023-12-06T15:49:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:49:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:49:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:49:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 13f99eeaee1a 2023-12-06T15:49:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.13f99eeaee1a 2023-12-06T15:49:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:49:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:49:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:49:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:49:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:49:51 2023 2023-12-06T15:49:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:49:51 2023 2023-12-06T15:49:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:49:51 2023 2023-12-06T15:49:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:49:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:49:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 3f1f4f6b-ba96-4370-b170-e42f93596565 2023-12-06T15:49:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:49:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:49:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_38 2023-12-06T15:49:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:49:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:49:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=7 2023-12-06T15:49:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:49:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:49:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 32318464 bytes 2023-12-06T15:49:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 125492224 bytes 2023-12-06T15:49:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 214743040 bytes 2023-12-06T15:49:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 234285056 bytes 2023-12-06T15:49:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 294482944 bytes 2023-12-06T15:49:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 295096320 bytes 2023-12-06T15:49:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 315395072 bytes 2023-12-06T15:49:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 370624512 bytes 2023-12-06T15:49:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 375449600 bytes 2023-12-06T15:49:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 481727488 bytes 2023-12-06T15:49:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 495611904 bytes 2023-12-06T15:49:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 506263552 bytes 2023-12-06T15:49:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 541076480 bytes 2023-12-06T15:49:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 622965760 bytes 2023-12-06T15:49:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 697456640 bytes 2023-12-06T15:49:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 763502592 bytes 2023-12-06T15:49:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 788153344 bytes 2023-12-06T15:49:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 815344640 bytes 2023-12-06T15:49:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 848976896 bytes 2023-12-06T15:49:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 876112896 bytes 2023-12-06T15:49:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 929766400 bytes 2023-12-06T15:49:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 966852608 bytes 2023-12-06T15:49:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1035640832 bytes 2023-12-06T15:49:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1068493824 bytes 2023-12-06T15:49:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1119360000 bytes 2023-12-06T15:49:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1177504768 bytes 2023-12-06T15:49:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1239591936 bytes 2023-12-06T15:49:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1257736192 bytes 2023-12-06T15:49:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1310560256 bytes 2023-12-06T15:49:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1341902848 bytes 2023-12-06T15:49:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1406311424 bytes 2023-12-06T15:49:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1431973888 bytes 2023-12-06T15:49:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1432586240 bytes 2023-12-06T15:49:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1522820096 bytes 2023-12-06T15:49:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1600016384 bytes 2023-12-06T15:49:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:49:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:49:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:49:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:49:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:49:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_39 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_39 128M 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:49:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:49:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:49:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:49:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:49:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2023-12-06T15:49:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:49:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-12-06T15:49:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:49:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:49:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:49:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:49:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-12-06T15:50:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:50:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-12-06T15:50:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_39': 2023-12-06T15:50:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:50:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:50:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:50:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 1405ae278316 2023-12-06T15:50:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.1405ae278316 2023-12-06T15:50:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:50:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:50:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:50:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:50:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:49:59 2023 2023-12-06T15:50:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:49:59 2023 2023-12-06T15:50:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:49:59 2023 2023-12-06T15:50:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:50:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:50:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 70cf571a-cab9-45c9-bf8a-20c249f7dcf1 2023-12-06T15:50:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:50:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:50:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_39 2023-12-06T15:50:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:50:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:50:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=10 2023-12-06T15:50:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:50:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:50:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 87320576 bytes 2023-12-06T15:50:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 204981248 bytes 2023-12-06T15:50:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 244371456 bytes 2023-12-06T15:50:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 278181888 bytes 2023-12-06T15:50:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 327774208 bytes 2023-12-06T15:50:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 368420864 bytes 2023-12-06T15:50:01.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 379179008 bytes 2023-12-06T15:50:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 394566656 bytes 2023-12-06T15:50:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 395194368 bytes 2023-12-06T15:50:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 438708224 bytes 2023-12-06T15:50:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 465822720 bytes 2023-12-06T15:50:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 498004992 bytes 2023-12-06T15:50:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 558951424 bytes 2023-12-06T15:50:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 585871360 bytes 2023-12-06T15:50:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 621876224 bytes 2023-12-06T15:50:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 674160640 bytes 2023-12-06T15:50:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 709485568 bytes 2023-12-06T15:50:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 745972736 bytes 2023-12-06T15:50:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 765497344 bytes 2023-12-06T15:50:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 766108672 bytes 2023-12-06T15:50:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 843279360 bytes 2023-12-06T15:50:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 857235456 bytes 2023-12-06T15:50:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 857853952 bytes 2023-12-06T15:50:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 909638656 bytes 2023-12-06T15:50:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 969802752 bytes 2023-12-06T15:50:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1025029120 bytes 2023-12-06T15:50:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1058411520 bytes 2023-12-06T15:50:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1102438400 bytes 2023-12-06T15:50:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1159468032 bytes 2023-12-06T15:50:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1223414784 bytes 2023-12-06T15:50:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1303515136 bytes 2023-12-06T15:50:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1325952000 bytes 2023-12-06T15:50:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1395773440 bytes 2023-12-06T15:50:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1436837888 bytes 2023-12-06T15:50:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1447038976 bytes 2023-12-06T15:50:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1452429312 bytes 2023-12-06T15:50:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1479366656 bytes 2023-12-06T15:50:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1555398656 bytes 2023-12-06T15:50:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1628847104 bytes 2023-12-06T15:50:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1670704128 bytes 2023-12-06T15:50:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1725363200 bytes 2023-12-06T15:50:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1771346944 bytes 2023-12-06T15:50:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1805507584 bytes 2023-12-06T15:50:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1858088960 bytes 2023-12-06T15:50:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1882664960 bytes 2023-12-06T15:50:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1918238720 bytes 2023-12-06T15:50:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1923751936 bytes 2023-12-06T15:50:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1937203200 bytes 2023-12-06T15:50:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1992805376 bytes 2023-12-06T15:50:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2009219072 bytes 2023-12-06T15:50:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2055117824 bytes 2023-12-06T15:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_40 2023-12-06T15:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-12-06T15:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_40 128M 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2023-12-06T15:50:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:50:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-12-06T15:50:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:50:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:50:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-12-06T15:50:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:50:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-12-06T15:50:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_40': 2023-12-06T15:50:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:50:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:50:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:50:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 141118584dfe 2023-12-06T15:50:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.141118584dfe 2023-12-06T15:50:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:50:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:50:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:50:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:50:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:50:10 2023 2023-12-06T15:50:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:50:10 2023 2023-12-06T15:50:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:50:10 2023 2023-12-06T15:50:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:50:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:50:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: bcd56cda-90d7-45de-b6e1-3c9183852338 2023-12-06T15:50:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:50:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:50:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_40 2023-12-06T15:50:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:50:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:50:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=6 2023-12-06T15:50:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:50:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:50:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 10218496 bytes 2023-12-06T15:50:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 90528768 bytes 2023-12-06T15:50:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 92096512 bytes 2023-12-06T15:50:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 100182016 bytes 2023-12-06T15:50:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 143380480 bytes 2023-12-06T15:50:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 213763072 bytes 2023-12-06T15:50:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 265995264 bytes 2023-12-06T15:50:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 296278016 bytes 2023-12-06T15:50:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 335224832 bytes 2023-12-06T15:50:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 392248320 bytes 2023-12-06T15:50:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 453361664 bytes 2023-12-06T15:50:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 524155904 bytes 2023-12-06T15:50:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 543980544 bytes 2023-12-06T15:50:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 593855488 bytes 2023-12-06T15:50:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 626374656 bytes 2023-12-06T15:50:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 724727808 bytes 2023-12-06T15:50:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 757301248 bytes 2023-12-06T15:50:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 788725760 bytes 2023-12-06T15:50:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 848965632 bytes 2023-12-06T15:50:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 880947200 bytes 2023-12-06T15:50:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 902692864 bytes 2023-12-06T15:50:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 934633472 bytes 2023-12-06T15:50:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1019259904 bytes 2023-12-06T15:50:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1038242816 bytes 2023-12-06T15:50:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1081271296 bytes 2023-12-06T15:50:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1105077248 bytes 2023-12-06T15:50:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1133440000 bytes 2023-12-06T15:50:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1194912768 bytes 2023-12-06T15:50:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1239851008 bytes 2023-12-06T15:50:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_41 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_41 128M 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2023-12-06T15:50:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:50:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-12-06T15:50:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:50:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:50:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-12-06T15:50:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:50:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-12-06T15:50:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_41': 2023-12-06T15:50:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 141d59282efc 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.141d59282efc 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:50:17 2023 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:50:17 2023 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:50:17 2023 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:50:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:50:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: d19eb783-02bd-4211-9062-fbc83db629fb 2023-12-06T15:50:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:50:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:50:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_41 2023-12-06T15:50:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:50:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:50:18.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=6 2023-12-06T15:50:18.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:50:18.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:50:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 76776448 bytes 2023-12-06T15:50:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 126984192 bytes 2023-12-06T15:50:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 173151232 bytes 2023-12-06T15:50:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 246651904 bytes 2023-12-06T15:50:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 340151296 bytes 2023-12-06T15:50:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 377016320 bytes 2023-12-06T15:50:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 386757632 bytes 2023-12-06T15:50:20.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 431515648 bytes 2023-12-06T15:50:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 519698432 bytes 2023-12-06T15:50:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 543983616 bytes 2023-12-06T15:50:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 559900672 bytes 2023-12-06T15:50:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 596169728 bytes 2023-12-06T15:50:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 622524416 bytes 2023-12-06T15:50:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 647283712 bytes 2023-12-06T15:50:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 741528576 bytes 2023-12-06T15:50:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 742157312 bytes 2023-12-06T15:50:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 757496832 bytes 2023-12-06T15:50:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 771462144 bytes 2023-12-06T15:50:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 818086912 bytes 2023-12-06T15:50:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 850719744 bytes 2023-12-06T15:50:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 886006784 bytes 2023-12-06T15:50:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 903359488 bytes 2023-12-06T15:50:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 933936128 bytes 2023-12-06T15:50:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1030611968 bytes 2023-12-06T15:50:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1097273344 bytes 2023-12-06T15:50:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1211753472 bytes 2023-12-06T15:50:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1247442944 bytes 2023-12-06T15:50:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1266869248 bytes 2023-12-06T15:50:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1324523520 bytes 2023-12-06T15:50:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:50:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:50:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:50:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:50:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:50:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_42 2023-12-06T15:50:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-12-06T15:50:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_42 128M 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:50:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:50:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:50:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:50:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2023-12-06T15:50:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:50:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-12-06T15:50:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:50:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:50:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-12-06T15:50:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-12-06T15:50:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_42': 2023-12-06T15:50:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:50:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:50:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:50:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 142916f78608 2023-12-06T15:50:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.142916f78608 2023-12-06T15:50:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:50:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:50:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:50:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:50:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:50:24 2023 2023-12-06T15:50:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:50:24 2023 2023-12-06T15:50:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:50:24 2023 2023-12-06T15:50:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:50:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:50:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 4c2635cc-57ee-4102-bf76-ae809079b50d 2023-12-06T15:50:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:50:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:50:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_42 2023-12-06T15:50:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:50:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:50:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=11 2023-12-06T15:50:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:50:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:50:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 76051456 bytes 2023-12-06T15:50:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 76697600 bytes 2023-12-06T15:50:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 91233280 bytes 2023-12-06T15:50:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 141116416 bytes 2023-12-06T15:50:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 193035264 bytes 2023-12-06T15:50:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 234397696 bytes 2023-12-06T15:50:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 301921280 bytes 2023-12-06T15:50:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 385461248 bytes 2023-12-06T15:50:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 409566208 bytes 2023-12-06T15:50:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 464885760 bytes 2023-12-06T15:50:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 472796160 bytes 2023-12-06T15:50:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 539104256 bytes 2023-12-06T15:50:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 580655104 bytes 2023-12-06T15:50:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 606013440 bytes 2023-12-06T15:50:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 658137088 bytes 2023-12-06T15:50:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 705168384 bytes 2023-12-06T15:50:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 763235328 bytes 2023-12-06T15:50:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 833290240 bytes 2023-12-06T15:50:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 876293120 bytes 2023-12-06T15:50:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 962621440 bytes 2023-12-06T15:50:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 982511616 bytes 2023-12-06T15:50:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1070737408 bytes 2023-12-06T15:50:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1093454848 bytes 2023-12-06T15:50:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1115715584 bytes 2023-12-06T15:50:30.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1166695424 bytes 2023-12-06T15:50:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1239252992 bytes 2023-12-06T15:50:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1253071872 bytes 2023-12-06T15:50:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1262364672 bytes 2023-12-06T15:50:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1277659136 bytes 2023-12-06T15:50:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1300105216 bytes 2023-12-06T15:50:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1338673152 bytes 2023-12-06T15:50:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1345490944 bytes 2023-12-06T15:50:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1409521664 bytes 2023-12-06T15:50:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1440783360 bytes 2023-12-06T15:50:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1495759872 bytes 2023-12-06T15:50:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1552269312 bytes 2023-12-06T15:50:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1625577472 bytes 2023-12-06T15:50:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1693266944 bytes 2023-12-06T15:50:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1730806784 bytes 2023-12-06T15:50:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1769533440 bytes 2023-12-06T15:50:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1783963648 bytes 2023-12-06T15:50:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1810041856 bytes 2023-12-06T15:50:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1824833536 bytes 2023-12-06T15:50:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1866365952 bytes 2023-12-06T15:50:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1947486208 bytes 2023-12-06T15:50:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1979123712 bytes 2023-12-06T15:50:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2017992704 bytes 2023-12-06T15:50:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2072398848 bytes 2023-12-06T15:50:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2085954560 bytes 2023-12-06T15:50:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2123779072 bytes 2023-12-06T15:50:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2133760000 bytes 2023-12-06T15:50:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2153324544 bytes 2023-12-06T15:50:35.326 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-06T15:50:35.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-06T15:50:35.327 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-06T15:50:35.327 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-06T15:50:35.327 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:50:35.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:50:35.326+0000 7fd1ffc01700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 119407) UID: 1000 2023-12-06T15:50:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2204124160 bytes 2023-12-06T15:50:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2204750848 bytes 2023-12-06T15:50:35.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:50:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2257763328 bytes 2023-12-06T15:50:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2314253312 bytes 2023-12-06T15:50:36.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2331348992 bytes 2023-12-06T15:50:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:50:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_43 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_43 128M 2023-12-06T15:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:50:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:50:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:50:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:50:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2023-12-06T15:50:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:50:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-12-06T15:50:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:50:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:50:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-12-06T15:50:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:50:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-12-06T15:50:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_43': 2023-12-06T15:50:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:50:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 1434372c689 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.1434372c689 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:50:36 2023 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:50:36 2023 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:50:36 2023 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: b9035ac9-e202-49a4-90ed-1f7ed899a231 2023-12-06T15:50:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:50:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:50:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_43 2023-12-06T15:50:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:50:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:50:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=9 2023-12-06T15:50:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:50:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:50:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 92492800 bytes 2023-12-06T15:50:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 162618368 bytes 2023-12-06T15:50:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 183025664 bytes 2023-12-06T15:50:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 240776192 bytes 2023-12-06T15:50:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 304893952 bytes 2023-12-06T15:50:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 345972736 bytes 2023-12-06T15:50:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 448014336 bytes 2023-12-06T15:50:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 517866496 bytes 2023-12-06T15:50:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 578152448 bytes 2023-12-06T15:50:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 580745216 bytes 2023-12-06T15:50:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 633791488 bytes 2023-12-06T15:50:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 680346624 bytes 2023-12-06T15:50:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 701934592 bytes 2023-12-06T15:50:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 726525952 bytes 2023-12-06T15:50:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 773371904 bytes 2023-12-06T15:50:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 803317760 bytes 2023-12-06T15:50:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 814768128 bytes 2023-12-06T15:50:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 873346048 bytes 2023-12-06T15:50:41.247 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-06T15:50:41.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-06T15:50:41.248 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-06T15:50:41.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:50:41.247+0000 7f5a401eb700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 132973) UID: 1000 2023-12-06T15:50:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 898586624 bytes 2023-12-06T15:50:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 920551424 bytes 2023-12-06T15:50:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:50:41.429 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-06T15:50:41.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-06T15:50:41.429 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-06T15:50:41.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-06T15:50:41.434 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2023-12-06T15:50:41.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:50:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 973294592 bytes 2023-12-06T15:50:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1006977024 bytes 2023-12-06T15:50:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1025611776 bytes 2023-12-06T15:50:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1042871296 bytes 2023-12-06T15:50:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1071979520 bytes 2023-12-06T15:50:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1172071424 bytes 2023-12-06T15:50:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1245809664 bytes 2023-12-06T15:50:43.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1375879168 bytes 2023-12-06T15:50:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1403399168 bytes 2023-12-06T15:50:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1440860160 bytes 2023-12-06T15:50:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1467693056 bytes 2023-12-06T15:50:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1511098368 bytes 2023-12-06T15:50:44.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1512912896 bytes 2023-12-06T15:50:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1562529792 bytes 2023-12-06T15:50:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1579234304 bytes 2023-12-06T15:50:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1602982912 bytes 2023-12-06T15:50:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1639128064 bytes 2023-12-06T15:50:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1698302976 bytes 2023-12-06T15:50:45.249 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-06T15:50:45.249 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:50:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:50:45.249 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-06T15:50:45.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-06T15:50:45.250 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-06T15:50:45.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-06T15:50:45.252 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2023-12-06T15:50:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1742033920 bytes 2023-12-06T15:50:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1873523712 bytes 2023-12-06T15:50:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1908287488 bytes 2023-12-06T15:50:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1928554496 bytes 2023-12-06T15:50:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1947596800 bytes 2023-12-06T15:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_44 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_44 128M 2023-12-06T15:50:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:50:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:50:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:50:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:50:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:50:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2023-12-06T15:50:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:50:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-12-06T15:50:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:50:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:50:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-12-06T15:50:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:50:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_44': 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 144f5b54d3e 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.144f5b54d3e 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:50:46 2023 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:50:46 2023 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:50:46 2023 2023-12-06T15:50:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:50:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:50:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 9a98f3b9-5d2f-447d-9e2f-0f9938f259fe 2023-12-06T15:50:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:50:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:50:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_44 2023-12-06T15:50:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:50:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:50:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:50:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:50:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=13 2023-12-06T15:50:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:50:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:50:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 27725824 bytes 2023-12-06T15:50:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 49820672 bytes 2023-12-06T15:50:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 93435904 bytes 2023-12-06T15:50:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 179578880 bytes 2023-12-06T15:50:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 272636928 bytes 2023-12-06T15:50:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 294852608 bytes 2023-12-06T15:50:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 374827008 bytes 2023-12-06T15:50:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 397708288 bytes 2023-12-06T15:50:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 437209088 bytes 2023-12-06T15:50:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 474297344 bytes 2023-12-06T15:50:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 544231424 bytes 2023-12-06T15:50:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 552428544 bytes 2023-12-06T15:50:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 599389184 bytes 2023-12-06T15:50:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 615349248 bytes 2023-12-06T15:50:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 633784320 bytes 2023-12-06T15:50:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 665519104 bytes 2023-12-06T15:50:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 705729536 bytes 2023-12-06T15:50:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 755066880 bytes 2023-12-06T15:50:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 763136000 bytes 2023-12-06T15:50:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 801883136 bytes 2023-12-06T15:50:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 885531648 bytes 2023-12-06T15:50:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 922426368 bytes 2023-12-06T15:50:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 931733504 bytes 2023-12-06T15:50:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 942059520 bytes 2023-12-06T15:50:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 965132288 bytes 2023-12-06T15:50:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 969305088 bytes 2023-12-06T15:50:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 973934592 bytes 2023-12-06T15:50:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1048464384 bytes 2023-12-06T15:50:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1093181440 bytes 2023-12-06T15:50:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1103307776 bytes 2023-12-06T15:50:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1151736832 bytes 2023-12-06T15:50:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1198282752 bytes 2023-12-06T15:50:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1236891648 bytes 2023-12-06T15:50:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1299778560 bytes 2023-12-06T15:50:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1332108288 bytes 2023-12-06T15:50:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1345720320 bytes 2023-12-06T15:50:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1346329600 bytes 2023-12-06T15:50:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1417968640 bytes 2023-12-06T15:50:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1486137344 bytes 2023-12-06T15:50:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1498806272 bytes 2023-12-06T15:50:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1538922496 bytes 2023-12-06T15:50:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1623988224 bytes 2023-12-06T15:50:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1678433280 bytes 2023-12-06T15:50:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1747653632 bytes 2023-12-06T15:50:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1759207424 bytes 2023-12-06T15:50:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1775606784 bytes 2023-12-06T15:50:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1783366656 bytes 2023-12-06T15:50:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1853205504 bytes 2023-12-06T15:50:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1896050688 bytes 2023-12-06T15:50:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1943659520 bytes 2023-12-06T15:50:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 1959725056 bytes 2023-12-06T15:50:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2022961152 bytes 2023-12-06T15:50:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2088341504 bytes 2023-12-06T15:50:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2201314304 bytes 2023-12-06T15:50:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2263902208 bytes 2023-12-06T15:50:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2277772288 bytes 2023-12-06T15:50:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2327867392 bytes 2023-12-06T15:50:58.500 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.498+0000 7f522af00700 -1 reset not still connected to 0x55eb28b501a0 2023-12-06T15:50:58.502 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.500+0000 7f522af00700 -1 reset not still connected to 0x55eb28b509c0 2023-12-06T15:50:58.502 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.500+0000 7f522af00700 -1 reset not still connected to 0x55eb28b50b60 2023-12-06T15:50:58.502 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.500+0000 7f522af00700 -1 reset not still connected to 0x55eb28b50dd0 2023-12-06T15:50:58.502 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.500+0000 7f522af00700 -1 reset not still connected to 0x55eb28b50f70 2023-12-06T15:50:58.503 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.500+0000 7f522af00700 -1 reset not still connected to 0x55eb28b51ad0 2023-12-06T15:50:58.503 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.500+0000 7f522af00700 -1 reset not still connected to 0x55eb28c9cea0 2023-12-06T15:50:58.503 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.500+0000 7f522af00700 -1 reset not still connected to 0x55eb30dc55f0 2023-12-06T15:50:58.503 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.500+0000 7f522af00700 -1 reset not still connected to 0x55eb30dc5a00 2023-12-06T15:50:58.503 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.500+0000 7f522af00700 -1 reset not still connected to 0x55eb313e64e0 2023-12-06T15:50:58.503 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.500+0000 7f522af00700 -1 reset not still connected to 0x55eb313e7ba0 2023-12-06T15:50:58.503 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.500+0000 7f522af00700 -1 reset not still connected to 0x55eb31a8be10 2023-12-06T15:50:58.503 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.500+0000 7f522af00700 -1 reset not still connected to 0x55eb31afa410 2023-12-06T15:50:58.503 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.501+0000 7f522af00700 -1 reset not still connected to 0x55eb32318750 2023-12-06T15:50:58.503 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.501+0000 7f522af00700 -1 reset not still connected to 0x55eb3251f2b0 2023-12-06T15:50:58.503 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.501+0000 7f522af00700 -1 reset not still connected to 0x55eb325bcc30 2023-12-06T15:50:58.504 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.501+0000 7f522af00700 -1 reset not still connected to 0x55eb325bd380 2023-12-06T15:50:58.504 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.501+0000 7f522af00700 -1 reset not still connected to 0x55eb325bd520 2023-12-06T15:50:58.504 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T15:50:58.501+0000 7f522af00700 -1 reset not still connected to 0x55eb3277bba0 2023-12-06T15:50:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2427491328 bytes 2023-12-06T15:50:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2475444224 bytes 2023-12-06T15:50:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2520541184 bytes 2023-12-06T15:50:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2574558208 bytes 2023-12-06T15:50:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2621517824 bytes 2023-12-06T15:50:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2642753536 bytes 2023-12-06T15:50:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2667829248 bytes 2023-12-06T15:51:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 2693257216 bytes 2023-12-06T15:51:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6600 IOs, 2733999104 bytes 2023-12-06T15:51:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:51:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:51:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_45 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_45 128M 2023-12-06T15:51:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:51:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:51:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:51:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:51:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2023-12-06T15:51:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:51:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-12-06T15:51:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:51:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:51:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-12-06T15:51:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:51:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-12-06T15:51:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_45': 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 145bed76f8ab 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.145bed76f8ab 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:51:00 2023 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:51:00 2023 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:51:00 2023 2023-12-06T15:51:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:51:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:51:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 58132a72-7f69-4342-8240-26753ebd27ff 2023-12-06T15:51:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:51:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:51:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_45 2023-12-06T15:51:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:51:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:51:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=11 2023-12-06T15:51:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:51:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:51:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 88398848 bytes 2023-12-06T15:51:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 107488256 bytes 2023-12-06T15:51:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 123094016 bytes 2023-12-06T15:51:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 181425152 bytes 2023-12-06T15:51:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 221083648 bytes 2023-12-06T15:51:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 237433856 bytes 2023-12-06T15:51:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 275015680 bytes 2023-12-06T15:51:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 320121856 bytes 2023-12-06T15:51:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 339204096 bytes 2023-12-06T15:51:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 392071168 bytes 2023-12-06T15:51:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 418236416 bytes 2023-12-06T15:51:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 441279488 bytes 2023-12-06T15:51:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 457556992 bytes 2023-12-06T15:51:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 531060736 bytes 2023-12-06T15:51:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 561532928 bytes 2023-12-06T15:51:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 576152576 bytes 2023-12-06T15:51:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 685243392 bytes 2023-12-06T15:51:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 740116480 bytes 2023-12-06T15:51:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 797733888 bytes 2023-12-06T15:51:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 834060288 bytes 2023-12-06T15:51:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 879916032 bytes 2023-12-06T15:51:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 903194624 bytes 2023-12-06T15:51:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 932469760 bytes 2023-12-06T15:51:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 975076352 bytes 2023-12-06T15:51:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 986013696 bytes 2023-12-06T15:51:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 998125568 bytes 2023-12-06T15:51:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 998767616 bytes 2023-12-06T15:51:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1044148224 bytes 2023-12-06T15:51:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1081677824 bytes 2023-12-06T15:51:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1094921216 bytes 2023-12-06T15:51:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1171377152 bytes 2023-12-06T15:51:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1217074176 bytes 2023-12-06T15:51:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1245800448 bytes 2023-12-06T15:51:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1345441792 bytes 2023-12-06T15:51:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1451023360 bytes 2023-12-06T15:51:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1473803264 bytes 2023-12-06T15:51:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1528758272 bytes 2023-12-06T15:51:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1569595392 bytes 2023-12-06T15:51:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1608995840 bytes 2023-12-06T15:51:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1644432384 bytes 2023-12-06T15:51:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1646394368 bytes 2023-12-06T15:51:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1676964864 bytes 2023-12-06T15:51:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1721903104 bytes 2023-12-06T15:51:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1751728128 bytes 2023-12-06T15:51:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1768892416 bytes 2023-12-06T15:51:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1804990464 bytes 2023-12-06T15:51:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1820062720 bytes 2023-12-06T15:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1879318528 bytes 2023-12-06T15:51:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1980831744 bytes 2023-12-06T15:51:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2034836480 bytes 2023-12-06T15:51:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2057327616 bytes 2023-12-06T15:51:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2094779392 bytes 2023-12-06T15:51:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2137381888 bytes 2023-12-06T15:51:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2185857024 bytes 2023-12-06T15:51:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2217085952 bytes 2023-12-06T15:51:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2284189696 bytes 2023-12-06T15:51:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:51:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:51:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:51:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:51:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:51:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_46 2023-12-06T15:51:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-12-06T15:51:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_46 128M 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:51:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:51:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:51:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2023-12-06T15:51:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:51:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-12-06T15:51:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:51:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:51:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-12-06T15:51:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:51:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-12-06T15:51:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_46': 2023-12-06T15:51:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:51:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:51:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:51:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 146770112479 2023-12-06T15:51:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.146770112479 2023-12-06T15:51:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:51:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:51:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:51:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:51:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:51:12 2023 2023-12-06T15:51:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:51:12 2023 2023-12-06T15:51:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:51:12 2023 2023-12-06T15:51:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:51:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:51:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c54283d5-014c-482e-9681-a7b875a0863c 2023-12-06T15:51:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:51:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:51:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_46 2023-12-06T15:51:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:51:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:51:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=6 2023-12-06T15:51:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:51:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:51:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 47556608 bytes 2023-12-06T15:51:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 48179200 bytes 2023-12-06T15:51:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 90305536 bytes 2023-12-06T15:51:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 128288768 bytes 2023-12-06T15:51:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 248128512 bytes 2023-12-06T15:51:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 253077504 bytes 2023-12-06T15:51:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 278133760 bytes 2023-12-06T15:51:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 306553856 bytes 2023-12-06T15:51:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 332037120 bytes 2023-12-06T15:51:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 383593472 bytes 2023-12-06T15:51:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 449752064 bytes 2023-12-06T15:51:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 470159360 bytes 2023-12-06T15:51:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 496296960 bytes 2023-12-06T15:51:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 514572288 bytes 2023-12-06T15:51:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 609449984 bytes 2023-12-06T15:51:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 635465728 bytes 2023-12-06T15:51:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 669917184 bytes 2023-12-06T15:51:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 670524416 bytes 2023-12-06T15:51:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 780185600 bytes 2023-12-06T15:51:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 805087232 bytes 2023-12-06T15:51:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 805684224 bytes 2023-12-06T15:51:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 847241216 bytes 2023-12-06T15:51:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 874699776 bytes 2023-12-06T15:51:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 931320832 bytes 2023-12-06T15:51:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 939540480 bytes 2023-12-06T15:51:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1012877312 bytes 2023-12-06T15:51:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1035455488 bytes 2023-12-06T15:51:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1075351552 bytes 2023-12-06T15:51:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1075971072 bytes 2023-12-06T15:51:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1126336512 bytes 2023-12-06T15:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1186663424 bytes 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_47 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:51:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_47 128M 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:51:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2023-12-06T15:51:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:51:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-12-06T15:51:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:51:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:51:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-12-06T15:51:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:51:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-12-06T15:51:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_47': 2023-12-06T15:51:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:51:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:51:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:51:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 1473243a86a1 2023-12-06T15:51:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.1473243a86a1 2023-12-06T15:51:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:51:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:51:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:51:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:51:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:51:19 2023 2023-12-06T15:51:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:51:19 2023 2023-12-06T15:51:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:51:19 2023 2023-12-06T15:51:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:51:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:51:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c4567031-4494-4727-b144-9c229376da8f 2023-12-06T15:51:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:51:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:51:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_47 2023-12-06T15:51:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:51:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:51:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=12 2023-12-06T15:51:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:51:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:51:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 64457728 bytes 2023-12-06T15:51:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 140681216 bytes 2023-12-06T15:51:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 167540736 bytes 2023-12-06T15:51:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 204797952 bytes 2023-12-06T15:51:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 228997120 bytes 2023-12-06T15:51:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 254794752 bytes 2023-12-06T15:51:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 299850752 bytes 2023-12-06T15:51:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 304363520 bytes 2023-12-06T15:51:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 320189440 bytes 2023-12-06T15:51:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 340962304 bytes 2023-12-06T15:51:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 348308480 bytes 2023-12-06T15:51:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 428966912 bytes 2023-12-06T15:51:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 462888960 bytes 2023-12-06T15:51:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 478153728 bytes 2023-12-06T15:51:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 526099456 bytes 2023-12-06T15:51:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 555501568 bytes 2023-12-06T15:51:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 600626176 bytes 2023-12-06T15:51:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 660057088 bytes 2023-12-06T15:51:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 685363200 bytes 2023-12-06T15:51:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 719036416 bytes 2023-12-06T15:51:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 743729152 bytes 2023-12-06T15:51:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 760374272 bytes 2023-12-06T15:51:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 787694592 bytes 2023-12-06T15:51:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 851397632 bytes 2023-12-06T15:51:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 856275968 bytes 2023-12-06T15:51:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 871398400 bytes 2023-12-06T15:51:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 908378112 bytes 2023-12-06T15:51:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 947445760 bytes 2023-12-06T15:51:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 975077376 bytes 2023-12-06T15:51:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1000665088 bytes 2023-12-06T15:51:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1035729920 bytes 2023-12-06T15:51:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1064846336 bytes 2023-12-06T15:51:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1092661248 bytes 2023-12-06T15:51:26.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1125068800 bytes 2023-12-06T15:51:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1128003584 bytes 2023-12-06T15:51:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1186743296 bytes 2023-12-06T15:51:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1241491456 bytes 2023-12-06T15:51:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1284347904 bytes 2023-12-06T15:51:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1308493824 bytes 2023-12-06T15:51:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1387895808 bytes 2023-12-06T15:51:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1411779584 bytes 2023-12-06T15:51:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1505335296 bytes 2023-12-06T15:51:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1513701376 bytes 2023-12-06T15:51:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1545606144 bytes 2023-12-06T15:51:28.583 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:51:28.581+0000 7f44ae2f8700 -1 reset not still connected to 0x559a0794d040 2023-12-06T15:51:28.583 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:51:28.581+0000 7f44ae2f8700 -1 reset not still connected to 0x559a14b7af70 2023-12-06T15:51:28.583 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:51:28.581+0000 7f44ae2f8700 -1 reset not still connected to 0x559a14c9cb60 2023-12-06T15:51:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1602420736 bytes 2023-12-06T15:51:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1607708672 bytes 2023-12-06T15:51:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1623487488 bytes 2023-12-06T15:51:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1663382528 bytes 2023-12-06T15:51:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1678518272 bytes 2023-12-06T15:51:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1693203456 bytes 2023-12-06T15:51:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 1750420480 bytes 2023-12-06T15:51:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 1788186624 bytes 2023-12-06T15:51:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 1839127552 bytes 2023-12-06T15:51:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 1840319488 bytes 2023-12-06T15:51:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 1857700864 bytes 2023-12-06T15:51:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 1884161024 bytes 2023-12-06T15:51:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 1902144512 bytes 2023-12-06T15:51:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 1969741824 bytes 2023-12-06T15:51:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2017017856 bytes 2023-12-06T15:51:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2096048128 bytes 2023-12-06T15:51:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2144868352 bytes 2023-12-06T15:51:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2209901568 bytes 2023-12-06T15:51:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:51:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_48 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_48 128M 2023-12-06T15:51:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:51:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:51:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:51:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:51:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2023-12-06T15:51:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:51:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-12-06T15:51:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:51:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:51:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-12-06T15:51:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:51:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-12-06T15:51:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_48': 2023-12-06T15:51:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:51:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:51:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:51:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 147f28a46bb1 2023-12-06T15:51:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.147f28a46bb1 2023-12-06T15:51:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:51:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:51:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:51:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:51:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:51:32 2023 2023-12-06T15:51:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:51:32 2023 2023-12-06T15:51:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:51:32 2023 2023-12-06T15:51:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:51:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:51:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 3ec3a26c-39fa-4d23-b6cb-9a34c415483f 2023-12-06T15:51:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:51:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:51:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_48 2023-12-06T15:51:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:51:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:51:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=8 2023-12-06T15:51:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:51:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:51:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 28641280 bytes 2023-12-06T15:51:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 58927104 bytes 2023-12-06T15:51:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 102434816 bytes 2023-12-06T15:51:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 137504768 bytes 2023-12-06T15:51:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 138120192 bytes 2023-12-06T15:51:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 175016960 bytes 2023-12-06T15:51:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 234766336 bytes 2023-12-06T15:51:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 245730304 bytes 2023-12-06T15:51:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 252255232 bytes 2023-12-06T15:51:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 277654528 bytes 2023-12-06T15:51:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 312763392 bytes 2023-12-06T15:51:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 351283200 bytes 2023-12-06T15:51:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 395763712 bytes 2023-12-06T15:51:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 442169344 bytes 2023-12-06T15:51:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 461369344 bytes 2023-12-06T15:51:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 509188096 bytes 2023-12-06T15:51:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 528292864 bytes 2023-12-06T15:51:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 554978304 bytes 2023-12-06T15:51:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 586074112 bytes 2023-12-06T15:51:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 643046400 bytes 2023-12-06T15:51:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 696853504 bytes 2023-12-06T15:51:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 723854336 bytes 2023-12-06T15:51:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 754964480 bytes 2023-12-06T15:51:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 767167488 bytes 2023-12-06T15:51:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 811017216 bytes 2023-12-06T15:51:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 827881472 bytes 2023-12-06T15:51:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 904969216 bytes 2023-12-06T15:51:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 911195136 bytes 2023-12-06T15:51:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 947652608 bytes 2023-12-06T15:51:39.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 952795136 bytes 2023-12-06T15:51:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 971538432 bytes 2023-12-06T15:51:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 987341824 bytes 2023-12-06T15:51:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1018238976 bytes 2023-12-06T15:51:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1038720000 bytes 2023-12-06T15:51:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1060216832 bytes 2023-12-06T15:51:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1092001792 bytes 2023-12-06T15:51:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1143355392 bytes 2023-12-06T15:51:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1253773312 bytes 2023-12-06T15:51:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1277302784 bytes 2023-12-06T15:51:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1326264320 bytes 2023-12-06T15:51:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1351829504 bytes 2023-12-06T15:51:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1408578560 bytes 2023-12-06T15:51:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1416403968 bytes 2023-12-06T15:51:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1450125312 bytes 2023-12-06T15:51:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:51:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:51:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:51:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:51:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:51:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_49 2023-12-06T15:51:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-12-06T15:51:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_49 128M 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:51:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2023-12-06T15:51:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:51:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-12-06T15:51:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:51:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:51:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-12-06T15:51:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:51:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-12-06T15:51:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_49': 2023-12-06T15:51:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:51:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:51:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:51:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 148be7456d00 2023-12-06T15:51:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.148be7456d00 2023-12-06T15:51:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:51:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:51:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:51:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:51:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:51:41 2023 2023-12-06T15:51:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:51:41 2023 2023-12-06T15:51:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:51:41 2023 2023-12-06T15:51:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:51:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:51:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: a5b2b9b1-e05d-4ea2-ba09-82cd6039f259 2023-12-06T15:51:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:51:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:51:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_49 2023-12-06T15:51:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:51:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:51:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=13 2023-12-06T15:51:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:51:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:51:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 86606848 bytes 2023-12-06T15:51:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 148524032 bytes 2023-12-06T15:51:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 160652288 bytes 2023-12-06T15:51:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 202431488 bytes 2023-12-06T15:51:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 238314496 bytes 2023-12-06T15:51:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 238917632 bytes 2023-12-06T15:51:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 282781696 bytes 2023-12-06T15:51:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 363271168 bytes 2023-12-06T15:51:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 422321152 bytes 2023-12-06T15:51:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 471861248 bytes 2023-12-06T15:51:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 543845376 bytes 2023-12-06T15:51:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 579118080 bytes 2023-12-06T15:51:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 612250624 bytes 2023-12-06T15:51:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 625904640 bytes 2023-12-06T15:51:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 693968896 bytes 2023-12-06T15:51:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 717805568 bytes 2023-12-06T15:51:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 748133376 bytes 2023-12-06T15:51:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 813666304 bytes 2023-12-06T15:51:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 849273856 bytes 2023-12-06T15:51:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 865137664 bytes 2023-12-06T15:51:46.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 865738752 bytes 2023-12-06T15:51:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 937738240 bytes 2023-12-06T15:51:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 958001152 bytes 2023-12-06T15:51:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 977473536 bytes 2023-12-06T15:51:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1022790656 bytes 2023-12-06T15:51:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1047742464 bytes 2023-12-06T15:51:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1073544192 bytes 2023-12-06T15:51:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1075837952 bytes 2023-12-06T15:51:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1127631872 bytes 2023-12-06T15:51:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1149407232 bytes 2023-12-06T15:51:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1156977664 bytes 2023-12-06T15:51:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1229618176 bytes 2023-12-06T15:51:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1264151552 bytes 2023-12-06T15:51:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1291638784 bytes 2023-12-06T15:51:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1384005632 bytes 2023-12-06T15:51:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1464604672 bytes 2023-12-06T15:51:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1504789504 bytes 2023-12-06T15:51:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1583799296 bytes 2023-12-06T15:51:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1612467200 bytes 2023-12-06T15:51:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1653322752 bytes 2023-12-06T15:51:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1696787456 bytes 2023-12-06T15:51:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1742144512 bytes 2023-12-06T15:51:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1814159360 bytes 2023-12-06T15:51:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1865909248 bytes 2023-12-06T15:51:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1930743808 bytes 2023-12-06T15:51:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1953445888 bytes 2023-12-06T15:51:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1980460032 bytes 2023-12-06T15:51:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2050138112 bytes 2023-12-06T15:51:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2101732352 bytes 2023-12-06T15:51:52.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2148657152 bytes 2023-12-06T15:51:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2208278528 bytes 2023-12-06T15:51:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2271782912 bytes 2023-12-06T15:51:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2309992448 bytes 2023-12-06T15:51:53.544 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:51:53.542+0000 7f44ae2f8700 -1 reset not still connected to 0x559a181a5e10 2023-12-06T15:51:53.544 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:51:53.542+0000 7f44ae2f8700 -1 reset not still connected to 0x559a1d69f2b0 2023-12-06T15:51:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2390307840 bytes 2023-12-06T15:51:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2418193408 bytes 2023-12-06T15:51:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2437863424 bytes 2023-12-06T15:51:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2438465536 bytes 2023-12-06T15:51:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2499063808 bytes 2023-12-06T15:51:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2537736192 bytes 2023-12-06T15:51:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2562010112 bytes 2023-12-06T15:51:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2595074048 bytes 2023-12-06T15:51:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2620651520 bytes 2023-12-06T15:51:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2730022912 bytes 2023-12-06T15:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2770063360 bytes 2023-12-06T15:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 2798688256 bytes 2023-12-06T15:51:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:51:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_50 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n snapshot ']' 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_50 128M 2023-12-06T15:51:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:51:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T15:51:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128M ']' 2023-12-06T15:51:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128M 2023-12-06T15:51:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T15:51:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2023-12-06T15:51:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' image = image ']' 2023-12-06T15:51:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-12-06T15:51:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:51:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T15:51:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-12-06T15:51:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T15:51:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_50': 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 14973daf3760 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.14973daf3760 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:51:55 2023 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:51:55 2023 2023-12-06T15:51:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:51:55 2023 2023-12-06T15:51:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:51:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:51:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c9d745c5-f598-4e49-aa04-df5e0f8e27d8 2023-12-06T15:51:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T15:51:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2023-12-06T15:51:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_50 2023-12-06T15:51:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:51:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:51:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:51:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T15:51:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=12 2023-12-06T15:51:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T15:51:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-12-06T15:51:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 20994048 bytes 2023-12-06T15:51:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 79035392 bytes 2023-12-06T15:51:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 93859840 bytes 2023-12-06T15:51:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 118510592 bytes 2023-12-06T15:51:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 171788288 bytes 2023-12-06T15:51:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 196171776 bytes 2023-12-06T15:51:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 253843456 bytes 2023-12-06T15:51:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 337882112 bytes 2023-12-06T15:51:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 387196928 bytes 2023-12-06T15:51:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 423565312 bytes 2023-12-06T15:51:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 459269120 bytes 2023-12-06T15:51:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 478777344 bytes 2023-12-06T15:51:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 503231488 bytes 2023-12-06T15:51:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 544808960 bytes 2023-12-06T15:51:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 601060352 bytes 2023-12-06T15:51:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 663898112 bytes 2023-12-06T15:51:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 704772096 bytes 2023-12-06T15:51:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 730977280 bytes 2023-12-06T15:51:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 817446912 bytes 2023-12-06T15:52:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 851253248 bytes 2023-12-06T15:52:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 887801856 bytes 2023-12-06T15:52:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 911391744 bytes 2023-12-06T15:52:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 999987200 bytes 2023-12-06T15:52:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1022348288 bytes 2023-12-06T15:52:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1108132864 bytes 2023-12-06T15:52:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1144911872 bytes 2023-12-06T15:52:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1164389376 bytes 2023-12-06T15:52:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1223717888 bytes 2023-12-06T15:52:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1247244288 bytes 2023-12-06T15:52:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1272748032 bytes 2023-12-06T15:52:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1335503872 bytes 2023-12-06T15:52:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1391634432 bytes 2023-12-06T15:52:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1405856768 bytes 2023-12-06T15:52:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1446798336 bytes 2023-12-06T15:52:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1479524352 bytes 2023-12-06T15:52:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1525044224 bytes 2023-12-06T15:52:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1548214272 bytes 2023-12-06T15:52:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1576735744 bytes 2023-12-06T15:52:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1617050624 bytes 2023-12-06T15:52:03.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1629175808 bytes 2023-12-06T15:52:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1663147008 bytes 2023-12-06T15:52:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1744169984 bytes 2023-12-06T15:52:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1799329792 bytes 2023-12-06T15:52:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1828052992 bytes 2023-12-06T15:52:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1837466624 bytes 2023-12-06T15:52:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1840186368 bytes 2023-12-06T15:52:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1851250688 bytes 2023-12-06T15:52:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1896965120 bytes 2023-12-06T15:52:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1939536896 bytes 2023-12-06T15:52:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1981932544 bytes 2023-12-06T15:52:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2058991616 bytes 2023-12-06T15:52:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2083529728 bytes 2023-12-06T15:52:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2100348928 bytes 2023-12-06T15:52:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2130238464 bytes 2023-12-06T15:52:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2147202048 bytes 2023-12-06T15:52:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2199405568 bytes 2023-12-06T15:52:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2234275840 bytes 2023-12-06T15:52:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2284951552 bytes 2023-12-06T15:52:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2311163904 bytes 2023-12-06T15:52:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2347788288 bytes 2023-12-06T15:52:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2436112384 bytes 2023-12-06T15:52:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2499878912 bytes 2023-12-06T15:52:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T15:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T15:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T15:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-12-06T15:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_count=50 2023-12-06T15:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count 2023-12-06T15:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_count=0 2023-12-06T15:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 40 2023-12-06T15:52:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in `seq 1 40` 2023-12-06T15:52:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 -ne 1 2023-12-06T15:52:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-06T15:52:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep 'images: ' 2023-12-06T15:52:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut '-d ' -f 2 2023-12-06T15:52:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:52:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ count=50 2023-12-06T15:52:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 50 = 50 2023-12-06T15:52:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-12-06T15:52:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:52:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state 2023-12-06T15:52:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 40 2023-12-06T15:52:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in `seq 1 40` 2023-12-06T15:52:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 -ne 1 2023-12-06T15:52:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-06T15:52:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep 'image health:' 2023-12-06T15:52:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut '-d ' -f 3 2023-12-06T15:52:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:52:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=OK 2023-12-06T15:52:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test OK = ERROR 2023-12-06T15:52:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test OK = OK 2023-12-06T15:52:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-12-06T15:52:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_count=50 2023-12-06T15:52:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:52:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count 2023-12-06T15:52:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_count=0 2023-12-06T15:52:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:52:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 40 2023-12-06T15:52:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in `seq 1 40` 2023-12-06T15:52:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 -ne 1 2023-12-06T15:52:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-06T15:52:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep 'images: ' 2023-12-06T15:52:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut '-d ' -f 2 2023-12-06T15:52:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:52:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ count=50 2023-12-06T15:52:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 50 = 50 2023-12-06T15:52:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-12-06T15:52:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:52:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state 2023-12-06T15:52:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 40 2023-12-06T15:52:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in `seq 1 40` 2023-12-06T15:52:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 -ne 1 2023-12-06T15:52:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-06T15:52:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep 'image health:' 2023-12-06T15:52:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut '-d ' -f 3 2023-12-06T15:52:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:52:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=WARNING 2023-12-06T15:52:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test WARNING = ERROR 2023-12-06T15:52:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test WARNING = OK 2023-12-06T15:52:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in `seq 1 40` 2023-12-06T15:52:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 -ne 1 2023-12-06T15:52:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 30 2023-12-06T15:52:32.435 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-06T15:52:32.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-06T15:52:32.436 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-06T15:52:32.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:52:32.435+0000 7f3f6319e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 133821) UID: 1000 2023-12-06T15:52:32.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:52:35.437 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-06T15:52:35.437 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:52:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:52:35.438 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-06T15:52:35.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-06T15:52:35.438 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-06T15:52:35.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-06T15:52:35.441 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2023-12-06T15:52:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-06T15:52:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep 'image health:' 2023-12-06T15:52:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut '-d ' -f 3 2023-12-06T15:52:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T15:52:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=OK 2023-12-06T15:52:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test OK = ERROR 2023-12-06T15:52:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test OK = OK 2023-12-06T15:52:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: compare many images' 2023-12-06T15:52:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.Q1BJHHgE3s/rbd-mirror.test.log 2023-12-06T15:52:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2023-12-06T15:52:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2023-12-06 15:52:38 TEST: compare many images 2023-12-06T15:52:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2023-12-06 15:52:38 TEST: compare many images 2023-12-06T15:52:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 50 2023-12-06T15:52:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:52:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_1 2023-12-06T15:52:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_1 snap 2023-12-06T15:52:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:52:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:52:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-12-06T15:52:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-12-06T15:52:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:52:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:52:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:52:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:52:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-12-06T15:52:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:52:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:52:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:52:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-12-06T15:52:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:52:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:52:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:52:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:52:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:52:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:52:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:52:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:52:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:52:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:52:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:52:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-12-06T15:52:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:52:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:52:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:52:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:52:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:52:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:52:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:52:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:52:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:52:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:52:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:52:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:52:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:52:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:52:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:52:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:52:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:52:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:52:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:52:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-12-06T15:52:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2023-12-06T15:52:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:52:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:52:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:52:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2023-12-06T15:52:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:52:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:52:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:52:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:52:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:52:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:52:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:52:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:52:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:52:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:52:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:52:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:52:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:52:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:52:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:52:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2023-12-06T15:52:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:52:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:52:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:52:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:52:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:52:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:52:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:52:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-12-06T15:52:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_1.status 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_1.status 2023-12-06T15:52:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:52:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_1.status 2023-12-06T15:52:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_1.status 2023-12-06T15:52:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_1.status 2023-12-06T15:52:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:52:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.status 2023-12-06T15:52:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-12-06T15:52:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-12-06T15:52:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 119 2023-12-06T15:52:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_1.status 2023-12-06T15:52:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_1.status 2023-12-06T15:52:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-12-06T15:52:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:52:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=119 2023-12-06T15:52:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:52:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:52:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:52:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.status 2023-12-06T15:52:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.status 2023-12-06T15:52:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-06T15:52:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:52:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=68 2023-12-06T15:52:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 119 = 68 2023-12-06T15:52:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:52:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:52:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.status 2023-12-06T15:52:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.status 2023-12-06T15:52:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-06T15:52:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:52:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:52:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:52:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:52:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:52:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:52:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 119 = '' 2023-12-06T15:52:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:52:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:52:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.status 2023-12-06T15:52:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.status 2023-12-06T15:52:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-06T15:52:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:52:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:52:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:52:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:52:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:52:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:52:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 119 = '' 2023-12-06T15:52:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:52:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:52:44.253 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-06T15:52:44.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-06T15:52:44.254 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-06T15:52:44.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:52:44.253+0000 7f8069697700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 133941) UID: 1000 2023-12-06T15:52:44.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:52:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.status 2023-12-06T15:52:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.status 2023-12-06T15:52:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-06T15:52:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:52:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=119 2023-12-06T15:52:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 119 = 119 2023-12-06T15:52:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-12-06T15:52:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:52:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:52:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:52:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:52:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-12-06T15:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_1': 2023-12-06T15:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:52:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2c325fdeb 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2c325fdeb 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:43:46 2023 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:43:46 2023 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:43:46 2023 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 0b9a1c1c-7b41-4db7-9c1f-9f9fa3190aa6 2023-12-06T15:52:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:52:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_1 snap 2023-12-06T15:52:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:52:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:52:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:52:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_1.export 2023-12-06T15:52:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.export 2023-12-06T15:52:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_1.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.export 2023-12-06T15:52:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_1.export 2023-12-06T15:52:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:52:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.export 2023-12-06T15:52:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:52:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_1.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.export 2023-12-06T15:52:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_1.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_1.export 2023-12-06T15:52:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:52:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_2 2023-12-06T15:52:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_2 snap 2023-12-06T15:52:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:52:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:52:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-12-06T15:52:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-12-06T15:52:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:52:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:52:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:52:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:52:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-12-06T15:52:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:52:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:52:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-12-06T15:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:52:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:52:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-06T15:52:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:52:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:52:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:52:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:52:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:52:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:52:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:52:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:52:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:52:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:52:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:52:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:52:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:52:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:52:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:52:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:52:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:52:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-06T15:52:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-06T15:52:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:52:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-12-06T15:52:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:52:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:52:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:52:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:52:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:52:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:52:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:52:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:52:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:52:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:52:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:52:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:52:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:52:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:52:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:52:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:52:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:52:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-12-06T15:52:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:52:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:52:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:52:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:52:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:52:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:52:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:52:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-12-06T15:52:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:52:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:52:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:52:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-12-06T15:52:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:52:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_2.status 2023-12-06T15:52:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_2.status 2023-12-06T15:52:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:52:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_2.status 2023-12-06T15:52:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_2.status 2023-12-06T15:52:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_2.status 2023-12-06T15:52:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:52:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.status 2023-12-06T15:52:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-12-06T15:52:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-12-06T15:52:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 121 2023-12-06T15:52:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_2.status 2023-12-06T15:52:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:50.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:50.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:50.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_2.status 2023-12-06T15:52:50.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-12-06T15:52:50.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:52:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=121 2023-12-06T15:52:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:52:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:52:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:52:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.status 2023-12-06T15:52:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.status 2023-12-06T15:52:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-06T15:52:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:52:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=69 2023-12-06T15:52:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 121 = 69 2023-12-06T15:52:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:52:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:52:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.status 2023-12-06T15:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.status 2023-12-06T15:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-06T15:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:52:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:52:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:52:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:52:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:52:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:52:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 121 = '' 2023-12-06T15:52:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:52:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:52:51.257 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-06T15:52:51.257 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:52:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:52:51.258 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-06T15:52:51.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-06T15:52:51.258 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-06T15:52:51.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-06T15:52:51.261 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2023-12-06T15:52:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.status 2023-12-06T15:52:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.status 2023-12-06T15:52:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-06T15:52:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:52:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:52:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:52:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:52:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:52:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:52:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 121 = '' 2023-12-06T15:52:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:52:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:52:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.status 2023-12-06T15:52:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.status 2023-12-06T15:52:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-06T15:52:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:52:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=121 2023-12-06T15:52:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 121 = 121 2023-12-06T15:52:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-12-06T15:52:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:52:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:52:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:52:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:52:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-12-06T15:52:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_2': 2023-12-06T15:52:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:52:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:52:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:52:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b240046d25 2023-12-06T15:52:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b240046d25 2023-12-06T15:52:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:52:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:52:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:52:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:52:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:43:58 2023 2023-12-06T15:52:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:43:58 2023 2023-12-06T15:52:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:43:58 2023 2023-12-06T15:52:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:52:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:52:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:52:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: ddc26196-98fe-465d-a33a-3c9e85520f7d 2023-12-06T15:52:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:52:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_2 snap 2023-12-06T15:52:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:52:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:52:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:52:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_2.export 2023-12-06T15:52:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.export 2023-12-06T15:52:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_2.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.export 2023-12-06T15:52:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_2.export 2023-12-06T15:52:54.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:52:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.export 2023-12-06T15:52:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:52:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_2.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.export 2023-12-06T15:52:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_2.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_2.export 2023-12-06T15:52:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:52:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_3 2023-12-06T15:52:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_3 snap 2023-12-06T15:52:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:52:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:52:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:52:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-12-06T15:52:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:52:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-12-06T15:52:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:52:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:52:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:52:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:52:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:52:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-12-06T15:52:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:52:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:52:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:52:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:52:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2023-12-06T15:52:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:52:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:52:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:52:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:52:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:52:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:52:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:52:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:52:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:52:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:52:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:52:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-12-06T15:52:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:52:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:52:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:52:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:52:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:52:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:52:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:52:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:52:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:52:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:52:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:52:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:52:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:52:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:52:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:52:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:52:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:52:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:52:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:52:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:52:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-12-06T15:52:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2023-12-06T15:52:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:52:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:52:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:52:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:52:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:52:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:52:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:52:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:52:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:52:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:52:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2023-12-06T15:52:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:52:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:52:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:52:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:52:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:52:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:52:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_3.status 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_3.status 2023-12-06T15:52:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:52:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_3.status 2023-12-06T15:52:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_3.status 2023-12-06T15:52:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_3.status 2023-12-06T15:52:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:52:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.status 2023-12-06T15:52:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-12-06T15:52:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:52:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-12-06T15:52:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 123 2023-12-06T15:52:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_3.status 2023-12-06T15:52:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:52:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:52:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_3.status 2023-12-06T15:52:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-12-06T15:52:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:52:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=123 2023-12-06T15:52:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:52:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:52:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:52:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.status 2023-12-06T15:52:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:52:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.status 2023-12-06T15:52:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-06T15:52:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:52:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=70 2023-12-06T15:52:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 123 = 70 2023-12-06T15:52:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:52:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:52:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.status 2023-12-06T15:52:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:52:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.status 2023-12-06T15:52:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-06T15:52:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:52:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:52:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:52:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:52:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:52:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:52:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 123 = '' 2023-12-06T15:52:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:52:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:52:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.status 2023-12-06T15:52:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:52:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:52:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:52:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.status 2023-12-06T15:52:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-06T15:52:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:52:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:52:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:52:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:52:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:52:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:52:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:52:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 123 = '' 2023-12-06T15:52:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:52:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:53:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.status 2023-12-06T15:53:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:53:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.status 2023-12-06T15:53:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-06T15:53:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=123 2023-12-06T15:53:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 123 = 123 2023-12-06T15:53:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-12-06T15:53:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:53:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:53:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-12-06T15:53:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_3': 2023-12-06T15:53:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:53:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:53:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:53:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b259e7ad35 2023-12-06T15:53:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b259e7ad35 2023-12-06T15:53:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:53:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:53:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:53:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:53:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:08 2023 2023-12-06T15:53:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:08 2023 2023-12-06T15:53:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:08 2023 2023-12-06T15:53:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:53:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:53:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:53:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 82a11cbd-ed20-4f4a-8948-aac80e8e0a67 2023-12-06T15:53:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:53:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_3 snap 2023-12-06T15:53:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:53:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:53:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_3.export 2023-12-06T15:53:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.export 2023-12-06T15:53:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_3.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.export 2023-12-06T15:53:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_3.export 2023-12-06T15:53:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.export 2023-12-06T15:53:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_3.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.export 2023-12-06T15:53:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_3.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_3.export 2023-12-06T15:53:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:53:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_4 2023-12-06T15:53:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_4 snap 2023-12-06T15:53:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:53:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-12-06T15:53:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-12-06T15:53:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:53:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-12-06T15:53:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:53:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:53:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:53:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-12-06T15:53:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:53:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:53:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:53:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:53:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:53:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:53:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:53:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:53:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-06T15:53:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:53:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:53:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:53:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:53:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:53:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:53:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:53:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-06T15:53:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-06T15:53:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:53:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:53:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-06T15:53:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:53:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:53:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:53:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-06T15:53:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:53:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:53:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:53:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:53:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:53:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:53:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:53:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-12-06T15:53:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:53:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:53:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-12-06T15:53:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_4.status 2023-12-06T15:53:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_4.status 2023-12-06T15:53:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_4.status 2023-12-06T15:53:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_4.status 2023-12-06T15:53:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_4.status 2023-12-06T15:53:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.status 2023-12-06T15:53:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-12-06T15:53:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-12-06T15:53:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 125 2023-12-06T15:53:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_4.status 2023-12-06T15:53:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_4.status 2023-12-06T15:53:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-12-06T15:53:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:53:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=125 2023-12-06T15:53:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:53:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.status 2023-12-06T15:53:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.status 2023-12-06T15:53:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-06T15:53:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=71 2023-12-06T15:53:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 125 = 71 2023-12-06T15:53:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:53:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.status 2023-12-06T15:53:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.status 2023-12-06T15:53:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-06T15:53:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 125 = '' 2023-12-06T15:53:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:53:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.status 2023-12-06T15:53:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.status 2023-12-06T15:53:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-06T15:53:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 125 = '' 2023-12-06T15:53:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:53:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.status 2023-12-06T15:53:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.status 2023-12-06T15:53:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-06T15:53:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=125 2023-12-06T15:53:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 125 = 125 2023-12-06T15:53:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-12-06T15:53:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:53:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-12-06T15:53:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_4': 2023-12-06T15:53:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:53:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b287ed83b4 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b287ed83b4 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:13 2023 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:13 2023 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:13 2023 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:53:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: b7f20ec4-f4b1-4e5b-a31f-db50e7727e1c 2023-12-06T15:53:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:53:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_4 snap 2023-12-06T15:53:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:53:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:53:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_4.export 2023-12-06T15:53:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.export 2023-12-06T15:53:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_4.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.export 2023-12-06T15:53:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_4.export 2023-12-06T15:53:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.export 2023-12-06T15:53:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_4.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.export 2023-12-06T15:53:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_4.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_4.export 2023-12-06T15:53:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:53:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_5 2023-12-06T15:53:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_5 snap 2023-12-06T15:53:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:53:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-12-06T15:53:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-12-06T15:53:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:53:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-12-06T15:53:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:53:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:53:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-12-06T15:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:53:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:53:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:53:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:53:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:53:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:53:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:53:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:53:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:53:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-12-06T15:53:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-06T15:53:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:53:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:53:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2023-12-06T15:53:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:53:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:53:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:53:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:53:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:53:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:53:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2023-12-06T15:53:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:53:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:53:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:53:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:53:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:53:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:53:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:53:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-12-06T15:53:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:53:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:53:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-12-06T15:53:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_5.status 2023-12-06T15:53:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_5.status 2023-12-06T15:53:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_5.status 2023-12-06T15:53:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_5.status 2023-12-06T15:53:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_5.status 2023-12-06T15:53:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.status 2023-12-06T15:53:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-12-06T15:53:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-12-06T15:53:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 127 2023-12-06T15:53:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_5.status 2023-12-06T15:53:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_5.status 2023-12-06T15:53:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-12-06T15:53:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:53:15.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=127 2023-12-06T15:53:15.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:15.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:53:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.status 2023-12-06T15:53:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.status 2023-12-06T15:53:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-06T15:53:15.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=72 2023-12-06T15:53:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 127 = 72 2023-12-06T15:53:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:53:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.status 2023-12-06T15:53:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.status 2023-12-06T15:53:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-06T15:53:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 127 = '' 2023-12-06T15:53:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:53:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.status 2023-12-06T15:53:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.status 2023-12-06T15:53:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-06T15:53:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 127 = '' 2023-12-06T15:53:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:53:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.status 2023-12-06T15:53:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.status 2023-12-06T15:53:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-06T15:53:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=127 2023-12-06T15:53:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 127 = 127 2023-12-06T15:53:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-12-06T15:53:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:53:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-12-06T15:53:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_5': 2023-12-06T15:53:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:53:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:53:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:53:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2c0256f86 2023-12-06T15:53:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2c0256f86 2023-12-06T15:53:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:53:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:53:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:53:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:53:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:23 2023 2023-12-06T15:53:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:23 2023 2023-12-06T15:53:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:23 2023 2023-12-06T15:53:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:53:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:53:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:53:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 292f59e4-56d9-46a7-8aff-49cef92c6c9c 2023-12-06T15:53:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:53:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_5 snap 2023-12-06T15:53:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:53:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:53:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_5.export 2023-12-06T15:53:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.export 2023-12-06T15:53:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_5.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.export 2023-12-06T15:53:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_5.export 2023-12-06T15:53:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.export 2023-12-06T15:53:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_5.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.export 2023-12-06T15:53:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_5.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_5.export 2023-12-06T15:53:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:53:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_6 2023-12-06T15:53:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_6 snap 2023-12-06T15:53:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:53:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-12-06T15:53:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-12-06T15:53:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:53:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-12-06T15:53:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:53:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:53:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:53:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2023-12-06T15:53:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:53:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:53:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:53:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:53:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:53:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:53:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:53:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:53:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-12-06T15:53:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:53:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:53:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:53:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:53:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:53:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:53:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:53:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:53:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:53:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:53:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:53:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:53:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-12-06T15:53:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-12-06T15:53:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:53:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:53:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2023-12-06T15:53:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:53:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:53:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:53:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:53:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:53:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:53:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2023-12-06T15:53:22.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:53:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:53:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:53:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:53:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:53:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:53:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:53:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-12-06T15:53:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_6.status 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_6.status 2023-12-06T15:53:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_6.status 2023-12-06T15:53:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_6.status 2023-12-06T15:53:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_6.status 2023-12-06T15:53:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.status 2023-12-06T15:53:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-12-06T15:53:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-12-06T15:53:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 129 2023-12-06T15:53:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_6.status 2023-12-06T15:53:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_6.status 2023-12-06T15:53:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-12-06T15:53:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:53:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=129 2023-12-06T15:53:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:53:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.status 2023-12-06T15:53:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.status 2023-12-06T15:53:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-06T15:53:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=73 2023-12-06T15:53:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 129 = 73 2023-12-06T15:53:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:53:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.status 2023-12-06T15:53:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.status 2023-12-06T15:53:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-06T15:53:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 129 = '' 2023-12-06T15:53:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:53:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.status 2023-12-06T15:53:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.status 2023-12-06T15:53:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-06T15:53:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 129 = '' 2023-12-06T15:53:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:53:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.status 2023-12-06T15:53:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.status 2023-12-06T15:53:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-06T15:53:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=129 2023-12-06T15:53:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 129 = 129 2023-12-06T15:53:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-12-06T15:53:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:53:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-12-06T15:53:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_6': 2023-12-06T15:53:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:53:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:53:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:53:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b22e401ece 2023-12-06T15:53:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b22e401ece 2023-12-06T15:53:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:53:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:53:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:53:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:53:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:29 2023 2023-12-06T15:53:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:29 2023 2023-12-06T15:53:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:29 2023 2023-12-06T15:53:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:53:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:53:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:53:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 1b242e7d-b3bc-4380-95c1-3f0e519a2478 2023-12-06T15:53:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:53:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_6 snap 2023-12-06T15:53:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:53:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:53:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_6.export 2023-12-06T15:53:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.export 2023-12-06T15:53:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_6.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.export 2023-12-06T15:53:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_6.export 2023-12-06T15:53:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.export 2023-12-06T15:53:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_6.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.export 2023-12-06T15:53:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_6.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_6.export 2023-12-06T15:53:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:53:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_7 2023-12-06T15:53:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_7 snap 2023-12-06T15:53:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:53:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-12-06T15:53:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-12-06T15:53:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:53:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-12-06T15:53:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:53:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:53:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:53:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-12-06T15:53:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:53:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:53:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:53:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:53:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:53:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:53:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:53:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:53:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-12-06T15:53:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:53:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:53:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:53:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:53:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:53:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:53:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:53:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:53:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:53:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-12-06T15:53:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-06T15:53:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:53:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:53:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:53:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:53:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:53:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:53:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2023-12-06T15:53:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:53:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:53:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:53:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:53:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:53:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:53:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:53:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-12-06T15:53:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-12-06T15:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_7.status 2023-12-06T15:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_7.status 2023-12-06T15:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_7.status 2023-12-06T15:53:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_7.status 2023-12-06T15:53:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_7.status 2023-12-06T15:53:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.status 2023-12-06T15:53:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-12-06T15:53:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-12-06T15:53:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 131 2023-12-06T15:53:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_7.status 2023-12-06T15:53:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_7.status 2023-12-06T15:53:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-12-06T15:53:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:53:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=131 2023-12-06T15:53:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:53:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.status 2023-12-06T15:53:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.status 2023-12-06T15:53:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-06T15:53:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=74 2023-12-06T15:53:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 131 = 74 2023-12-06T15:53:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:53:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.status 2023-12-06T15:53:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.status 2023-12-06T15:53:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-06T15:53:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 131 = '' 2023-12-06T15:53:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:53:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.status 2023-12-06T15:53:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.status 2023-12-06T15:53:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-06T15:53:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 131 = '' 2023-12-06T15:53:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:53:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.status 2023-12-06T15:53:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.status 2023-12-06T15:53:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-06T15:53:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=131 2023-12-06T15:53:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 131 = 131 2023-12-06T15:53:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-12-06T15:53:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:53:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-12-06T15:53:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_7': 2023-12-06T15:53:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:53:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:53:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:53:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2732018a0 2023-12-06T15:53:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2732018a0 2023-12-06T15:53:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:53:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:53:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:53:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:53:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:34 2023 2023-12-06T15:53:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:34 2023 2023-12-06T15:53:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:34 2023 2023-12-06T15:53:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:53:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:53:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:53:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: a45295cc-1ac0-4902-a0d3-c086b4cca650 2023-12-06T15:53:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:53:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_7 snap 2023-12-06T15:53:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:53:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:53:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_7.export 2023-12-06T15:53:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.export 2023-12-06T15:53:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_7.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.export 2023-12-06T15:53:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_7.export 2023-12-06T15:53:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.export 2023-12-06T15:53:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_7.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.export 2023-12-06T15:53:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_7.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_7.export 2023-12-06T15:53:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:53:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_8 2023-12-06T15:53:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_8 snap 2023-12-06T15:53:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-12-06T15:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-12-06T15:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:53:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-12-06T15:53:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:53:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:53:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:53:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2023-12-06T15:53:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:53:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:53:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-12-06T15:53:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:53:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:53:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-12-06T15:53:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2023-12-06T15:53:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:53:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:53:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2023-12-06T15:53:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:53:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:53:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:53:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:53:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:53:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:53:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2023-12-06T15:53:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:53:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:53:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:53:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:53:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:53:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:53:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:53:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_8.status 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_8.status 2023-12-06T15:53:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_8.status 2023-12-06T15:53:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_8.status 2023-12-06T15:53:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_8.status 2023-12-06T15:53:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.status 2023-12-06T15:53:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-12-06T15:53:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-12-06T15:53:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 133 2023-12-06T15:53:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_8.status 2023-12-06T15:53:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_8.status 2023-12-06T15:53:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-12-06T15:53:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:53:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=133 2023-12-06T15:53:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:53:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.status 2023-12-06T15:53:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.status 2023-12-06T15:53:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-06T15:53:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=75 2023-12-06T15:53:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 133 = 75 2023-12-06T15:53:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:53:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.status 2023-12-06T15:53:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.status 2023-12-06T15:53:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-06T15:53:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 133 = '' 2023-12-06T15:53:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:53:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.status 2023-12-06T15:53:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.status 2023-12-06T15:53:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-06T15:53:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 133 = '' 2023-12-06T15:53:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:53:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.status 2023-12-06T15:53:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.status 2023-12-06T15:53:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-06T15:53:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=133 2023-12-06T15:53:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 133 = 133 2023-12-06T15:53:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-12-06T15:53:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:53:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_8': 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2f96a17 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2f96a17 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:48 2023 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:48 2023 2023-12-06T15:53:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:48 2023 2023-12-06T15:53:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:53:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:53:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:53:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f4484446-8808-413e-bb2b-96fb61446c98 2023-12-06T15:53:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:53:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_8 snap 2023-12-06T15:53:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:53:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:53:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_8.export 2023-12-06T15:53:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.export 2023-12-06T15:53:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_8.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.export 2023-12-06T15:53:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_8.export 2023-12-06T15:53:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.export 2023-12-06T15:53:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_8.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.export 2023-12-06T15:53:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_8.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_8.export 2023-12-06T15:53:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:53:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_9 2023-12-06T15:53:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_9 snap 2023-12-06T15:53:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:53:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-12-06T15:53:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-12-06T15:53:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:53:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-12-06T15:53:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:53:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:53:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:53:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-12-06T15:53:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:53:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:53:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:53:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:53:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:53:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:53:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:53:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:53:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-12-06T15:53:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:53:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:53:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:53:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:53:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:53:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:53:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:53:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-12-06T15:53:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-06T15:53:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:53:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:53:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2023-12-06T15:53:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:53:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:53:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:53:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:53:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:53:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:53:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:53:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2023-12-06T15:53:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:53:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:53:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:53:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:53:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:53:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_9.status 2023-12-06T15:53:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_9.status 2023-12-06T15:53:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_9.status 2023-12-06T15:53:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_9.status 2023-12-06T15:53:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_9.status 2023-12-06T15:53:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.status 2023-12-06T15:53:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-12-06T15:53:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-12-06T15:53:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 135 2023-12-06T15:53:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_9.status 2023-12-06T15:53:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_9.status 2023-12-06T15:53:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-12-06T15:53:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:53:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=135 2023-12-06T15:53:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:53:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.status 2023-12-06T15:53:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.status 2023-12-06T15:53:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-06T15:53:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=76 2023-12-06T15:53:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 135 = 76 2023-12-06T15:53:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:53:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.status 2023-12-06T15:53:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.status 2023-12-06T15:53:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-06T15:53:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 135 = '' 2023-12-06T15:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:53:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.status 2023-12-06T15:53:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.status 2023-12-06T15:53:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-06T15:53:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 135 = '' 2023-12-06T15:53:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:53:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.status 2023-12-06T15:53:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.status 2023-12-06T15:53:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-06T15:53:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=135 2023-12-06T15:53:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 135 = 135 2023-12-06T15:53:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-12-06T15:53:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:53:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-12-06T15:53:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_9': 2023-12-06T15:53:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:53:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:53:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:53:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2ed779cd4 2023-12-06T15:53:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2ed779cd4 2023-12-06T15:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:44:59 2023 2023-12-06T15:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:44:59 2023 2023-12-06T15:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:44:59 2023 2023-12-06T15:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f202fb70-81a7-49b9-b61d-6d5b735f390f 2023-12-06T15:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:53:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_9 snap 2023-12-06T15:53:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:53:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:53:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_9.export 2023-12-06T15:53:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.export 2023-12-06T15:53:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_9.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.export 2023-12-06T15:53:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_9.export 2023-12-06T15:53:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.export 2023-12-06T15:53:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:53:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_9.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.export 2023-12-06T15:53:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_9.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_9.export 2023-12-06T15:53:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:53:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_10 2023-12-06T15:53:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_10 snap 2023-12-06T15:53:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:53:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-12-06T15:53:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-12-06T15:53:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:53:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:53:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-12-06T15:53:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:53:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:53:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:53:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2023-12-06T15:53:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:53:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:53:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:53:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:53:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:53:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:53:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:53:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:53:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:53:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-12-06T15:53:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:53:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:53:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:53:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:53:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:53:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:53:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:53:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:53:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:53:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-12-06T15:53:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-06T15:53:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:53:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:53:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:53:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-12-06T15:53:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:53:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:53:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:53:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:53:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:53:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:53:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:53:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:53:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:53:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:53:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:53:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:53:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:53:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:53:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:53:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:53:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-12-06T15:53:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:53:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:53:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:53:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:53:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:53:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:53:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:53:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_10.status 2023-12-06T15:53:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_10.status 2023-12-06T15:53:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_10.status 2023-12-06T15:53:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_10.status 2023-12-06T15:53:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_10.status 2023-12-06T15:53:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:53:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.status 2023-12-06T15:53:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-12-06T15:53:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-12-06T15:53:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 137 2023-12-06T15:53:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_10.status 2023-12-06T15:53:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:53:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_10.status 2023-12-06T15:53:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-12-06T15:53:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:53:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=137 2023-12-06T15:53:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:53:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.status 2023-12-06T15:53:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.status 2023-12-06T15:53:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-06T15:53:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:56.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=77 2023-12-06T15:53:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 137 = 77 2023-12-06T15:53:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:53:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.status 2023-12-06T15:53:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.status 2023-12-06T15:53:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-06T15:53:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 137 = '' 2023-12-06T15:53:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:53:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.status 2023-12-06T15:53:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.status 2023-12-06T15:53:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-06T15:53:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:53:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:53:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:53:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:53:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:53:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 137 = '' 2023-12-06T15:53:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:53:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:53:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.status 2023-12-06T15:53:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.status 2023-12-06T15:53:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-06T15:53:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:53:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:53:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=137 2023-12-06T15:53:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 137 = 137 2023-12-06T15:53:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:53:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-12-06T15:53:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:53:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:53:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:53:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:53:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:53:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:53:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-12-06T15:54:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_10': 2023-12-06T15:54:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:54:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:54:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:54:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2f2e7fd32 2023-12-06T15:54:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2f2e7fd32 2023-12-06T15:54:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:54:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:54:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:54:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:54:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:45:06 2023 2023-12-06T15:54:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:45:06 2023 2023-12-06T15:54:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:45:06 2023 2023-12-06T15:54:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:54:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:54:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:54:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 70428fb0-ef2f-4a77-a9f3-f9c2d486736f 2023-12-06T15:54:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:54:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_10 snap 2023-12-06T15:54:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:54:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:54:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_10.export 2023-12-06T15:54:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.export 2023-12-06T15:54:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_10.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.export 2023-12-06T15:54:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_10.export 2023-12-06T15:54:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.export 2023-12-06T15:54:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_10.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.export 2023-12-06T15:54:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_10.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_10.export 2023-12-06T15:54:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:54:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_11 2023-12-06T15:54:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_11 snap 2023-12-06T15:54:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:54:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-12-06T15:54:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-12-06T15:54:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:54:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-12-06T15:54:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:54:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:54:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:54:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2023-12-06T15:54:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:54:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:54:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:54:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:54:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:54:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:54:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:54:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:54:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-12-06T15:54:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:54:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:54:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:54:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:54:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-12-06T15:54:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-06T15:54:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:54:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:54:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2023-12-06T15:54:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:54:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:54:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:54:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2023-12-06T15:54:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:54:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:54:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:54:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:54:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:54:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:54:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:54:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-12-06T15:54:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:54:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:54:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-12-06T15:54:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_11.status 2023-12-06T15:54:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_11.status 2023-12-06T15:54:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_11.status 2023-12-06T15:54:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_11.status 2023-12-06T15:54:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_11.status 2023-12-06T15:54:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.status 2023-12-06T15:54:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-12-06T15:54:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-12-06T15:54:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 139 2023-12-06T15:54:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_11.status 2023-12-06T15:54:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_11.status 2023-12-06T15:54:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-12-06T15:54:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:54:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=139 2023-12-06T15:54:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:54:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.status 2023-12-06T15:54:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.status 2023-12-06T15:54:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-06T15:54:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=78 2023-12-06T15:54:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 139 = 78 2023-12-06T15:54:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:54:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.status 2023-12-06T15:54:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.status 2023-12-06T15:54:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-06T15:54:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 139 = '' 2023-12-06T15:54:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:54:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.status 2023-12-06T15:54:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.status 2023-12-06T15:54:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-06T15:54:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 139 = '' 2023-12-06T15:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:54:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.status 2023-12-06T15:54:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.status 2023-12-06T15:54:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-06T15:54:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=139 2023-12-06T15:54:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 139 = 139 2023-12-06T15:54:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-12-06T15:54:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:54:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-12-06T15:54:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_11': 2023-12-06T15:54:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:54:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b278430686 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b278430686 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:45:15 2023 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:45:15 2023 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:45:15 2023 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 85642e31-6be7-4259-b3a5-09a8ae17198f 2023-12-06T15:54:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:54:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_11 snap 2023-12-06T15:54:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T15:54:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:54:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_11.export 2023-12-06T15:54:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.export 2023-12-06T15:54:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_11.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.export 2023-12-06T15:54:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_11.export 2023-12-06T15:54:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.export 2023-12-06T15:54:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_11.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.export 2023-12-06T15:54:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_11.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_11.export 2023-12-06T15:54:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:54:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_12 2023-12-06T15:54:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_12 snap 2023-12-06T15:54:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:54:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-12-06T15:54:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-12-06T15:54:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:54:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-12-06T15:54:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2023-12-06T15:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:54:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:54:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:54:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:54:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:54:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:54:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-06T15:54:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-06T15:54:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:54:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:54:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:54:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-06T15:54:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:54:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:54:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:54:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:54:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:54:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:54:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:54:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-12-06T15:54:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:54:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:54:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-12-06T15:54:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_12.status 2023-12-06T15:54:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_12.status 2023-12-06T15:54:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_12.status 2023-12-06T15:54:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_12.status 2023-12-06T15:54:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_12.status 2023-12-06T15:54:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.status 2023-12-06T15:54:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-12-06T15:54:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-12-06T15:54:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 141 2023-12-06T15:54:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_12.status 2023-12-06T15:54:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_12.status 2023-12-06T15:54:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-12-06T15:54:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:54:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=141 2023-12-06T15:54:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:54:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.status 2023-12-06T15:54:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.status 2023-12-06T15:54:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-06T15:54:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=79 2023-12-06T15:54:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 141 = 79 2023-12-06T15:54:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:54:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.status 2023-12-06T15:54:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.status 2023-12-06T15:54:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-06T15:54:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 141 = '' 2023-12-06T15:54:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:54:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.status 2023-12-06T15:54:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.status 2023-12-06T15:54:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-06T15:54:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 141 = '' 2023-12-06T15:54:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:54:15.441 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6469104190130195) > weight (0.25) 2023-12-06T15:54:15.441 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.699009037637384) > weight (0.25) 2023-12-06T15:54:15.442 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6151780662757361) > weight (0.25) 2023-12-06T15:54:15.442 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9744314297094293) > weight (0.25) 2023-12-06T15:54:15.442 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2023-12-06T15:54:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.status 2023-12-06T15:54:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.status 2023-12-06T15:54:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-06T15:54:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=141 2023-12-06T15:54:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 141 = 141 2023-12-06T15:54:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-12-06T15:54:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:54:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-12-06T15:54:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_12': 2023-12-06T15:54:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2fc46637 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2fc46637 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:45:26 2023 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:45:26 2023 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:45:26 2023 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:54:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 2bd42a12-9068-4ea7-b5df-749817b7f6cc 2023-12-06T15:54:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:54:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_12 snap 2023-12-06T15:54:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T15:54:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:54:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_12.export 2023-12-06T15:54:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.export 2023-12-06T15:54:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_12.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.export 2023-12-06T15:54:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_12.export 2023-12-06T15:54:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.export 2023-12-06T15:54:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_12.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.export 2023-12-06T15:54:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_12.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_12.export 2023-12-06T15:54:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:54:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_13 2023-12-06T15:54:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_13 snap 2023-12-06T15:54:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:54:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-12-06T15:54:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-12-06T15:54:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:54:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-12-06T15:54:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:54:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:54:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:54:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-12-06T15:54:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:54:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:54:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:54:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:54:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:54:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:54:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:54:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:54:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:54:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:54:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:54:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:54:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:54:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:54:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:54:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:54:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-12-06T15:54:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-12-06T15:54:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:54:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:54:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:54:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:54:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2023-12-06T15:54:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:54:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:54:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:54:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:54:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:54:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:54:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:54:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_13.status 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_13.status 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_13.status 2023-12-06T15:54:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_13.status 2023-12-06T15:54:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_13.status 2023-12-06T15:54:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.status 2023-12-06T15:54:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-12-06T15:54:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-12-06T15:54:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 143 2023-12-06T15:54:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_13.status 2023-12-06T15:54:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_13.status 2023-12-06T15:54:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-12-06T15:54:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:54:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=143 2023-12-06T15:54:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:54:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.status 2023-12-06T15:54:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.status 2023-12-06T15:54:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-06T15:54:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=80 2023-12-06T15:54:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 143 = 80 2023-12-06T15:54:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:54:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.status 2023-12-06T15:54:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.status 2023-12-06T15:54:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-06T15:54:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 143 = '' 2023-12-06T15:54:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:54:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.status 2023-12-06T15:54:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.status 2023-12-06T15:54:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-06T15:54:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 143 = '' 2023-12-06T15:54:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:54:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.status 2023-12-06T15:54:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.status 2023-12-06T15:54:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-06T15:54:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=143 2023-12-06T15:54:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 143 = 143 2023-12-06T15:54:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-12-06T15:54:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:54:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-12-06T15:54:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_13': 2023-12-06T15:54:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:54:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:54:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:54:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2fbaece0d 2023-12-06T15:54:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2fbaece0d 2023-12-06T15:54:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:54:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:54:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:54:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:54:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:45:37 2023 2023-12-06T15:54:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:45:37 2023 2023-12-06T15:54:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:45:37 2023 2023-12-06T15:54:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:54:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:54:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:54:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: d04aac8e-3190-4279-b4cb-4b1a37a6b42f 2023-12-06T15:54:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:54:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_13 snap 2023-12-06T15:54:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T15:54:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:54:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_13.export 2023-12-06T15:54:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.export 2023-12-06T15:54:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_13.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.export 2023-12-06T15:54:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_13.export 2023-12-06T15:54:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.export 2023-12-06T15:54:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_13.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.export 2023-12-06T15:54:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_13.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_13.export 2023-12-06T15:54:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:54:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_14 2023-12-06T15:54:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_14 snap 2023-12-06T15:54:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:54:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-12-06T15:54:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-12-06T15:54:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:54:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-12-06T15:54:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:54:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:54:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:54:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-12-06T15:54:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:54:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:54:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:54:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:54:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:54:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:54:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:54:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:54:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-12-06T15:54:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:54:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:54:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:54:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:54:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:54:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:54:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:54:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:54:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:54:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:54:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:54:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:54:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-12-06T15:54:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-06T15:54:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:54:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:54:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2023-12-06T15:54:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:54:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:54:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:54:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:54:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:54:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:54:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2023-12-06T15:54:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:54:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:54:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:54:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:54:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_14.status 2023-12-06T15:54:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_14.status 2023-12-06T15:54:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_14.status 2023-12-06T15:54:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_14.status 2023-12-06T15:54:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_14.status 2023-12-06T15:54:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.status 2023-12-06T15:54:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-12-06T15:54:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-12-06T15:54:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 145 2023-12-06T15:54:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_14.status 2023-12-06T15:54:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_14.status 2023-12-06T15:54:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-12-06T15:54:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:54:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=145 2023-12-06T15:54:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:54:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.status 2023-12-06T15:54:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.status 2023-12-06T15:54:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-06T15:54:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=81 2023-12-06T15:54:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 145 = 81 2023-12-06T15:54:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:54:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.status 2023-12-06T15:54:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.status 2023-12-06T15:54:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-06T15:54:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 145 = '' 2023-12-06T15:54:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:54:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.status 2023-12-06T15:54:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.status 2023-12-06T15:54:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-06T15:54:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 145 = '' 2023-12-06T15:54:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:54:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.status 2023-12-06T15:54:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.status 2023-12-06T15:54:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-06T15:54:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=145 2023-12-06T15:54:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 145 = 145 2023-12-06T15:54:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-12-06T15:54:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:54:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-12-06T15:54:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_14': 2023-12-06T15:54:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:54:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:54:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:54:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b26dde545c 2023-12-06T15:54:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b26dde545c 2023-12-06T15:54:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:45:51 2023 2023-12-06T15:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:45:51 2023 2023-12-06T15:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:45:51 2023 2023-12-06T15:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 216c840f-8878-4153-b225-e1d08de4c3fd 2023-12-06T15:54:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:54:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_14 snap 2023-12-06T15:54:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T15:54:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:54:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_14.export 2023-12-06T15:54:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.export 2023-12-06T15:54:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_14.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.export 2023-12-06T15:54:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_14.export 2023-12-06T15:54:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.export 2023-12-06T15:54:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_14.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.export 2023-12-06T15:54:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_14.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_14.export 2023-12-06T15:54:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:54:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_15 2023-12-06T15:54:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_15 snap 2023-12-06T15:54:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:54:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-12-06T15:54:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-12-06T15:54:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:54:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:36.261 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6656635037750835) > weight (0.25) 2023-12-06T15:54:36.261 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9778864721715338) > weight (0.25) 2023-12-06T15:54:36.261 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9593358728967147) > weight (0.25) 2023-12-06T15:54:36.262 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5195551152415647) > weight (0.25) 2023-12-06T15:54:36.262 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2023-12-06T15:54:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-12-06T15:54:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:54:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:54:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:54:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-12-06T15:54:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:54:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:54:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:54:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:54:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:54:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:54:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:54:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:54:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:54:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-12-06T15:54:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-12-06T15:54:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:54:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:54:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2023-12-06T15:54:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:54:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:54:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:54:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:54:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:54:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:54:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2023-12-06T15:54:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:54:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:54:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:54:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:54:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_15.status 2023-12-06T15:54:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_15.status 2023-12-06T15:54:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_15.status 2023-12-06T15:54:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_15.status 2023-12-06T15:54:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_15.status 2023-12-06T15:54:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.status 2023-12-06T15:54:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-12-06T15:54:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-12-06T15:54:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 147 2023-12-06T15:54:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_15.status 2023-12-06T15:54:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_15.status 2023-12-06T15:54:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-12-06T15:54:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:54:38.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=147 2023-12-06T15:54:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:54:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.status 2023-12-06T15:54:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.status 2023-12-06T15:54:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-06T15:54:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=82 2023-12-06T15:54:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 147 = 82 2023-12-06T15:54:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:54:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.status 2023-12-06T15:54:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.status 2023-12-06T15:54:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-06T15:54:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 147 = '' 2023-12-06T15:54:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:54:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.status 2023-12-06T15:54:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.status 2023-12-06T15:54:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-06T15:54:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 147 = '' 2023-12-06T15:54:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:54:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.status 2023-12-06T15:54:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.status 2023-12-06T15:54:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-06T15:54:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=147 2023-12-06T15:54:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 147 = 147 2023-12-06T15:54:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-12-06T15:54:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:54:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-12-06T15:54:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_15': 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b28b659aea 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b28b659aea 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:46:00 2023 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:46:00 2023 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:46:00 2023 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:54:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:54:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 500b919a-f976-4b7b-8c44-5adea4cfc7db 2023-12-06T15:54:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:54:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_15 snap 2023-12-06T15:54:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T15:54:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:54:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_15.export 2023-12-06T15:54:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.export 2023-12-06T15:54:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_15.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.export 2023-12-06T15:54:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_15.export 2023-12-06T15:54:42.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.export 2023-12-06T15:54:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_15.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.export 2023-12-06T15:54:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_15.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_15.export 2023-12-06T15:54:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:54:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_16 2023-12-06T15:54:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_16 snap 2023-12-06T15:54:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:54:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-12-06T15:54:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-12-06T15:54:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:54:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-12-06T15:54:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:54:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:54:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:54:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-12-06T15:54:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:54:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:54:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:54:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:54:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:54:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:54:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:54:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:54:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-12-06T15:54:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:54:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:54:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:54:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:54:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:54:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:54:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-12-06T15:54:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-06T15:54:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:54:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2023-12-06T15:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:54:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:54:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:54:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:54:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2023-12-06T15:54:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:54:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:54:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:54:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:54:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-12-06T15:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_16.status 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_16.status 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_16.status 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_16.status 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_16.status 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.status 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-12-06T15:54:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 149 2023-12-06T15:54:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_16.status 2023-12-06T15:54:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_16.status 2023-12-06T15:54:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-12-06T15:54:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:54:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=149 2023-12-06T15:54:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:54:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.status 2023-12-06T15:54:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.status 2023-12-06T15:54:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-06T15:54:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=83 2023-12-06T15:54:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 149 = 83 2023-12-06T15:54:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:54:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.status 2023-12-06T15:54:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.status 2023-12-06T15:54:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-06T15:54:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 149 = '' 2023-12-06T15:54:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:54:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.status 2023-12-06T15:54:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.status 2023-12-06T15:54:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-06T15:54:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 149 = '' 2023-12-06T15:54:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:54:47.884 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.878+0000 7f44ae2f8700 -1 reset not still connected to 0x559a05b86b60 2023-12-06T15:54:47.884 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.878+0000 7f44ae2f8700 -1 reset not still connected to 0x559a061325b0 2023-12-06T15:54:47.884 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.878+0000 7f44ae2f8700 -1 reset not still connected to 0x559a0617d930 2023-12-06T15:54:47.884 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.880+0000 7f44ae2f8700 -1 reset not still connected to 0x559a0626ec30 2023-12-06T15:54:47.884 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.880+0000 7f44ae2f8700 -1 reset not still connected to 0x559a069f0a90 2023-12-06T15:54:47.884 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.880+0000 7f44ae2f8700 -1 reset not still connected to 0x559a06ab6ea0 2023-12-06T15:54:47.884 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.880+0000 7f44ae2f8700 -1 reset not still connected to 0x559a06ab7ba0 2023-12-06T15:54:47.884 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.880+0000 7f44ae2f8700 -1 reset not still connected to 0x559a072c20d0 2023-12-06T15:54:47.884 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.880+0000 7f44ae2f8700 -1 reset not still connected to 0x559a0794d860 2023-12-06T15:54:47.884 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.880+0000 7f44ae2f8700 -1 reset not still connected to 0x559a0bcda750 2023-12-06T15:54:47.884 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.880+0000 7f44ae2f8700 -1 reset not still connected to 0x559a0bcdb110 2023-12-06T15:54:47.885 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.880+0000 7f44ae2f8700 -1 reset not still connected to 0x559a0bcdb520 2023-12-06T15:54:47.885 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.880+0000 7f44ae2f8700 -1 reset not still connected to 0x559a14c9c270 2023-12-06T15:54:47.885 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.880+0000 7f44ae2f8700 -1 reset not still connected to 0x559a151b4d00 2023-12-06T15:54:47.885 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.881+0000 7f44ae2f8700 -1 reset not still connected to 0x559a180d7450 2023-12-06T15:54:47.885 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.881+0000 7f44ae2f8700 -1 reset not still connected to 0x559a181a4dd0 2023-12-06T15:54:47.885 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.881+0000 7f44ae2f8700 -1 reset not still connected to 0x559a1d69e8f0 2023-12-06T15:54:47.885 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.882+0000 7f44ae2f8700 -1 reset not still connected to 0x559a20bd5380 2023-12-06T15:54:47.885 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.882+0000 7f44ae2f8700 -1 reset not still connected to 0x559a2171c340 2023-12-06T15:54:47.885 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.882+0000 7f44ae2f8700 -1 reset not still connected to 0x559a27fe5c70 2023-12-06T15:54:47.885 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:54:47.882+0000 7f44ae2f8700 -1 reset not still connected to 0x559a344dcdd0 2023-12-06T15:54:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.status 2023-12-06T15:54:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.status 2023-12-06T15:54:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-06T15:54:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=149 2023-12-06T15:54:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 149 = 149 2023-12-06T15:54:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-12-06T15:54:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:54:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-12-06T15:54:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_16': 2023-12-06T15:54:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:54:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:54:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:54:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b28ba2b135 2023-12-06T15:54:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b28ba2b135 2023-12-06T15:54:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:54:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:54:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:54:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:54:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:46:16 2023 2023-12-06T15:54:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:46:16 2023 2023-12-06T15:54:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:46:16 2023 2023-12-06T15:54:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:54:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:54:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:54:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: dc7f2a11-16c4-4259-8d8f-2015f551ea93 2023-12-06T15:54:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:54:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_16 snap 2023-12-06T15:54:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T15:54:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:54:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_16.export 2023-12-06T15:54:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.export 2023-12-06T15:54:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_16.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.export 2023-12-06T15:54:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_16.export 2023-12-06T15:54:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.export 2023-12-06T15:54:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_16.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.export 2023-12-06T15:54:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_16.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_16.export 2023-12-06T15:54:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:54:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_17 2023-12-06T15:54:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_17 snap 2023-12-06T15:54:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:54:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-12-06T15:54:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-12-06T15:54:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:54:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-12-06T15:54:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:54:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:54:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:54:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-12-06T15:54:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:54:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:54:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:54:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:54:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:54:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:54:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:54:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:54:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:54:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-12-06T15:54:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:54:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:54:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:54:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:54:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:54:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-12-06T15:54:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-12-06T15:54:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:54:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:54:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:54:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:54:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:54:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:54:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2023-12-06T15:54:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:54:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:54:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:54:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:54:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:54:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:54:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:54:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-12-06T15:54:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:54:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:54:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-12-06T15:54:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:54:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_17.status 2023-12-06T15:54:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_17.status 2023-12-06T15:54:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_17.status 2023-12-06T15:54:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_17.status 2023-12-06T15:54:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_17.status 2023-12-06T15:54:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:54:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.status 2023-12-06T15:54:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-12-06T15:54:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-12-06T15:54:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 151 2023-12-06T15:54:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_17.status 2023-12-06T15:54:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_17.status 2023-12-06T15:54:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-12-06T15:54:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:54:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=151 2023-12-06T15:54:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:54:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.status 2023-12-06T15:54:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.status 2023-12-06T15:54:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-06T15:54:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=84 2023-12-06T15:54:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 151 = 84 2023-12-06T15:54:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:54:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.status 2023-12-06T15:54:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.status 2023-12-06T15:54:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-06T15:54:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 151 = '' 2023-12-06T15:54:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:54:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.status 2023-12-06T15:54:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.status 2023-12-06T15:54:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-06T15:54:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:54:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:54:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:54:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:54:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:54:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 151 = '' 2023-12-06T15:54:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:54:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:54:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.status 2023-12-06T15:54:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.status 2023-12-06T15:54:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-06T15:54:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:54:58.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:54:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=151 2023-12-06T15:54:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 151 = 151 2023-12-06T15:54:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-12-06T15:54:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:54:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:54:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:54:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:54:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-12-06T15:54:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_17': 2023-12-06T15:54:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:54:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:54:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:54:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2cc57e61 2023-12-06T15:54:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2cc57e61 2023-12-06T15:54:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:54:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:54:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:54:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:54:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:46:26 2023 2023-12-06T15:54:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:46:26 2023 2023-12-06T15:54:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:46:26 2023 2023-12-06T15:54:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:54:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:54:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:54:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: b11d8c4b-991a-40fd-a328-c851d1d3afd6 2023-12-06T15:54:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:54:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_17 snap 2023-12-06T15:54:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T15:54:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:54:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_17.export 2023-12-06T15:54:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.export 2023-12-06T15:54:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_17.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.export 2023-12-06T15:54:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_17.export 2023-12-06T15:54:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.export 2023-12-06T15:54:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:54:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_17.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.export 2023-12-06T15:54:59.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_17.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_17.export 2023-12-06T15:54:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:54:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:54:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_18 2023-12-06T15:54:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_18 snap 2023-12-06T15:54:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:54:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:54:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:54:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:54:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:55:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-12-06T15:55:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-12-06T15:55:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:55:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-12-06T15:55:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:55:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:55:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:55:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2023-12-06T15:55:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:55:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:55:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:55:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:55:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:55:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:55:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:55:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:55:02.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:55:02.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:55:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-12-06T15:55:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-06T15:55:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:55:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:55:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2023-12-06T15:55:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:55:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:55:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:55:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:55:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:55:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:55:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2023-12-06T15:55:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:55:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:55:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:55:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:55:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_18.status 2023-12-06T15:55:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_18.status 2023-12-06T15:55:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_18.status 2023-12-06T15:55:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_18.status 2023-12-06T15:55:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_18.status 2023-12-06T15:55:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.status 2023-12-06T15:55:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-12-06T15:55:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-12-06T15:55:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 153 2023-12-06T15:55:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_18.status 2023-12-06T15:55:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_18.status 2023-12-06T15:55:03.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-12-06T15:55:03.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:55:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=153 2023-12-06T15:55:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:55:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.status 2023-12-06T15:55:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.status 2023-12-06T15:55:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-06T15:55:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=85 2023-12-06T15:55:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 153 = 85 2023-12-06T15:55:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:55:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.status 2023-12-06T15:55:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.status 2023-12-06T15:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-06T15:55:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 153 = '' 2023-12-06T15:55:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:55:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.status 2023-12-06T15:55:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.status 2023-12-06T15:55:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-06T15:55:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 153 = '' 2023-12-06T15:55:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:55:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.status 2023-12-06T15:55:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.status 2023-12-06T15:55:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-06T15:55:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=153 2023-12-06T15:55:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 153 = 153 2023-12-06T15:55:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-12-06T15:55:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:55:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-12-06T15:55:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_18': 2023-12-06T15:55:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:55:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:55:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:55:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2cb2032f8 2023-12-06T15:55:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2cb2032f8 2023-12-06T15:55:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:55:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:55:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:55:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:55:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:46:34 2023 2023-12-06T15:55:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:46:34 2023 2023-12-06T15:55:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:46:34 2023 2023-12-06T15:55:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:55:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:55:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:55:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: bf968f6f-af59-4018-a2bf-c1b130c0104d 2023-12-06T15:55:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:55:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_18 snap 2023-12-06T15:55:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T15:55:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:55:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_18.export 2023-12-06T15:55:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.export 2023-12-06T15:55:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_18.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.export 2023-12-06T15:55:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_18.export 2023-12-06T15:55:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.export 2023-12-06T15:55:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_18.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.export 2023-12-06T15:55:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_18.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_18.export 2023-12-06T15:55:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:55:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_19 2023-12-06T15:55:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_19 snap 2023-12-06T15:55:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:55:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-12-06T15:55:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-12-06T15:55:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:55:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-12-06T15:55:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:55:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:55:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:55:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-12-06T15:55:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:55:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:55:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:55:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:55:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:55:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:55:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:55:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:55:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-12-06T15:55:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:55:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:55:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-12-06T15:55:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-06T15:55:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:55:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:55:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2023-12-06T15:55:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:55:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:55:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:55:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:55:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:55:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2023-12-06T15:55:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:55:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:55:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:55:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:55:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_19.status 2023-12-06T15:55:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_19.status 2023-12-06T15:55:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_19.status 2023-12-06T15:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_19.status 2023-12-06T15:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_19.status 2023-12-06T15:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.status 2023-12-06T15:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-12-06T15:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-12-06T15:55:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 155 2023-12-06T15:55:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_19.status 2023-12-06T15:55:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_19.status 2023-12-06T15:55:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-12-06T15:55:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:55:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=155 2023-12-06T15:55:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:55:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.status 2023-12-06T15:55:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.status 2023-12-06T15:55:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-06T15:55:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 155 = '' 2023-12-06T15:55:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:55:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.status 2023-12-06T15:55:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.status 2023-12-06T15:55:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-06T15:55:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 155 = '' 2023-12-06T15:55:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:55:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.status 2023-12-06T15:55:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.status 2023-12-06T15:55:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-06T15:55:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 155 = '' 2023-12-06T15:55:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:55:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.status 2023-12-06T15:55:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.status 2023-12-06T15:55:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-06T15:55:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=155 2023-12-06T15:55:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 155 = 155 2023-12-06T15:55:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-12-06T15:55:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:55:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-12-06T15:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_19': 2023-12-06T15:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b23e2a5427 2023-12-06T15:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b23e2a5427 2023-12-06T15:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:46:46 2023 2023-12-06T15:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:46:46 2023 2023-12-06T15:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:46:46 2023 2023-12-06T15:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: cadad124-8d2e-48fe-a7b8-df7c14a3f2c0 2023-12-06T15:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:55:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_19 snap 2023-12-06T15:55:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T15:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_19.export 2023-12-06T15:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.export 2023-12-06T15:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_19.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.export 2023-12-06T15:55:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_19.export 2023-12-06T15:55:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.export 2023-12-06T15:55:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_19.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.export 2023-12-06T15:55:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_19.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_19.export 2023-12-06T15:55:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:55:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_20 2023-12-06T15:55:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_20 snap 2023-12-06T15:55:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-12-06T15:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-12-06T15:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-12-06T15:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-12-06T15:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:55:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:55:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:55:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:55:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:55:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:55:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:55:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:55:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-06T15:55:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-06T15:55:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:55:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:55:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-06T15:55:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:55:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:55:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:55:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:55:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:55:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:55:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-06T15:55:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:55:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:55:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:55:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:55:19.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:55:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:55:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:55:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-12-06T15:55:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:55:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:55:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-12-06T15:55:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_20.status 2023-12-06T15:55:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_20.status 2023-12-06T15:55:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_20.status 2023-12-06T15:55:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_20.status 2023-12-06T15:55:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_20.status 2023-12-06T15:55:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.status 2023-12-06T15:55:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-12-06T15:55:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-12-06T15:55:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 157 2023-12-06T15:55:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_20.status 2023-12-06T15:55:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_20.status 2023-12-06T15:55:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-12-06T15:55:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:55:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=157 2023-12-06T15:55:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:55:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.status 2023-12-06T15:55:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.status 2023-12-06T15:55:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-06T15:55:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=87 2023-12-06T15:55:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 157 = 87 2023-12-06T15:55:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:55:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.status 2023-12-06T15:55:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.status 2023-12-06T15:55:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-06T15:55:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 157 = '' 2023-12-06T15:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:55:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.status 2023-12-06T15:55:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.status 2023-12-06T15:55:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-06T15:55:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 157 = '' 2023-12-06T15:55:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:55:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.status 2023-12-06T15:55:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.status 2023-12-06T15:55:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-06T15:55:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=157 2023-12-06T15:55:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 157 = 157 2023-12-06T15:55:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-12-06T15:55:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:55:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:24.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-12-06T15:55:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_20': 2023-12-06T15:55:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:55:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:55:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:55:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2d2f49616 2023-12-06T15:55:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2d2f49616 2023-12-06T15:55:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:55:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:55:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:55:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:55:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:46:53 2023 2023-12-06T15:55:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:46:53 2023 2023-12-06T15:55:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:46:53 2023 2023-12-06T15:55:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:55:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:55:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:55:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 7c1ca29b-6451-4600-b10e-63d11596db15 2023-12-06T15:55:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:55:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_20 snap 2023-12-06T15:55:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T15:55:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:55:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_20.export 2023-12-06T15:55:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.export 2023-12-06T15:55:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_20.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.export 2023-12-06T15:55:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_20.export 2023-12-06T15:55:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.export 2023-12-06T15:55:24.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_20.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.export 2023-12-06T15:55:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_20.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_20.export 2023-12-06T15:55:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:55:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_21 2023-12-06T15:55:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_21 snap 2023-12-06T15:55:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:55:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-12-06T15:55:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-12-06T15:55:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:55:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-12-06T15:55:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:55:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:55:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:55:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-12-06T15:55:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:55:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:55:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:55:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:55:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:55:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:55:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:55:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:55:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:55:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:55:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:55:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:55:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:55:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:55:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:55:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-12-06T15:55:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-12-06T15:55:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:55:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:55:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:55:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:55:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:55:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2023-12-06T15:55:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:55:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:55:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:55:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:55:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:55:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:55:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:55:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-12-06T15:55:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:55:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:55:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-12-06T15:55:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_21.status 2023-12-06T15:55:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_21.status 2023-12-06T15:55:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_21.status 2023-12-06T15:55:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_21.status 2023-12-06T15:55:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_21.status 2023-12-06T15:55:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.status 2023-12-06T15:55:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-12-06T15:55:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-12-06T15:55:28.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 159 2023-12-06T15:55:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_21.status 2023-12-06T15:55:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_21.status 2023-12-06T15:55:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-12-06T15:55:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:55:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=159 2023-12-06T15:55:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:55:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.status 2023-12-06T15:55:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.status 2023-12-06T15:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-06T15:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=88 2023-12-06T15:55:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 159 = 88 2023-12-06T15:55:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:55:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.status 2023-12-06T15:55:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.status 2023-12-06T15:55:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-06T15:55:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 159 = '' 2023-12-06T15:55:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:55:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.status 2023-12-06T15:55:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.status 2023-12-06T15:55:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-06T15:55:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 159 = '' 2023-12-06T15:55:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:55:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.status 2023-12-06T15:55:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.status 2023-12-06T15:55:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-06T15:55:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=159 2023-12-06T15:55:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 159 = 159 2023-12-06T15:55:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-12-06T15:55:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:55:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-12-06T15:55:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_21': 2023-12-06T15:55:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:55:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:55:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:55:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b283325724 2023-12-06T15:55:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b283325724 2023-12-06T15:55:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:55:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:55:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:55:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:55:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:05 2023 2023-12-06T15:55:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:05 2023 2023-12-06T15:55:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:05 2023 2023-12-06T15:55:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:55:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:55:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:55:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 298c48a0-cc50-45ca-995e-2df71ee446f2 2023-12-06T15:55:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:55:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_21 snap 2023-12-06T15:55:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T15:55:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:55:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_21.export 2023-12-06T15:55:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.export 2023-12-06T15:55:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_21.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.export 2023-12-06T15:55:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_21.export 2023-12-06T15:55:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.export 2023-12-06T15:55:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_21.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.export 2023-12-06T15:55:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_21.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_21.export 2023-12-06T15:55:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:55:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_22 2023-12-06T15:55:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_22 snap 2023-12-06T15:55:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:55:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-12-06T15:55:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-12-06T15:55:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:55:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:55:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:55:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:55:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:55:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-12-06T15:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:55:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-12-06T15:55:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-06T15:55:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:55:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:55:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2023-12-06T15:55:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:55:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:55:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:55:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:55:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:55:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:55:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2023-12-06T15:55:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:55:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:55:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:55:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:55:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:55:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:55:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:55:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-12-06T15:55:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:55:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:55:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-12-06T15:55:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_22.status 2023-12-06T15:55:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_22.status 2023-12-06T15:55:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_22.status 2023-12-06T15:55:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_22.status 2023-12-06T15:55:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_22.status 2023-12-06T15:55:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.status 2023-12-06T15:55:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-12-06T15:55:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-12-06T15:55:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 161 2023-12-06T15:55:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_22.status 2023-12-06T15:55:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_22.status 2023-12-06T15:55:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-12-06T15:55:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:55:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=161 2023-12-06T15:55:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:55:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.status 2023-12-06T15:55:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.status 2023-12-06T15:55:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-06T15:55:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=89 2023-12-06T15:55:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 161 = 89 2023-12-06T15:55:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:55:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.status 2023-12-06T15:55:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.status 2023-12-06T15:55:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-06T15:55:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 161 = '' 2023-12-06T15:55:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:55:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.status 2023-12-06T15:55:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.status 2023-12-06T15:55:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-06T15:55:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 161 = '' 2023-12-06T15:55:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:55:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.status 2023-12-06T15:55:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.status 2023-12-06T15:55:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-06T15:55:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=161 2023-12-06T15:55:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 161 = 161 2023-12-06T15:55:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-12-06T15:55:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:55:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-12-06T15:55:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_22': 2023-12-06T15:55:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:55:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:55:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:55:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2f3fd4525 2023-12-06T15:55:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2f3fd4525 2023-12-06T15:55:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:55:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:55:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:55:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:55:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:18 2023 2023-12-06T15:55:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:18 2023 2023-12-06T15:55:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:18 2023 2023-12-06T15:55:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:55:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:55:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:55:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: f9078af2-0a9f-4ece-8c12-87bbc54a7d15 2023-12-06T15:55:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:55:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_22 snap 2023-12-06T15:55:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T15:55:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:55:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_22.export 2023-12-06T15:55:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.export 2023-12-06T15:55:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_22.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.export 2023-12-06T15:55:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_22.export 2023-12-06T15:55:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.export 2023-12-06T15:55:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_22.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.export 2023-12-06T15:55:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_22.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_22.export 2023-12-06T15:55:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:55:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_23 2023-12-06T15:55:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_23 snap 2023-12-06T15:55:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:55:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-12-06T15:55:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-12-06T15:55:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:55:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-12-06T15:55:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:55:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:55:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:55:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-12-06T15:55:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:55:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:55:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:55:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:55:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:55:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:55:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:55:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:55:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:55:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:55:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-12-06T15:55:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-06T15:55:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:55:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:55:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2023-12-06T15:55:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:55:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:55:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:55:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:55:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:55:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:55:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:55:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:55:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:55:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:55:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:55:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:55:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2023-12-06T15:55:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:55:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:55:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:55:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:55:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:55:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:55:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:55:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-12-06T15:55:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:55:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:55:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-12-06T15:55:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_23.status 2023-12-06T15:55:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_23.status 2023-12-06T15:55:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_23.status 2023-12-06T15:55:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_23.status 2023-12-06T15:55:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_23.status 2023-12-06T15:55:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.status 2023-12-06T15:55:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-12-06T15:55:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-12-06T15:55:44.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 163 2023-12-06T15:55:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_23.status 2023-12-06T15:55:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_23.status 2023-12-06T15:55:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-12-06T15:55:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:55:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=163 2023-12-06T15:55:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:55:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.status 2023-12-06T15:55:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.status 2023-12-06T15:55:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-06T15:55:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=90 2023-12-06T15:55:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 163 = 90 2023-12-06T15:55:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:55:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.status 2023-12-06T15:55:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.status 2023-12-06T15:55:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-06T15:55:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 163 = '' 2023-12-06T15:55:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:55:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.status 2023-12-06T15:55:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.status 2023-12-06T15:55:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-06T15:55:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 163 = '' 2023-12-06T15:55:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:55:46.262 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-06T15:55:46.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-06T15:55:46.263 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-06T15:55:46.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:55:46.261+0000 7f64c2618700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 135234) UID: 1000 2023-12-06T15:55:46.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:55:47.263 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-06T15:55:47.264 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:55:47.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:55:47.264 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-06T15:55:47.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-06T15:55:47.265 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-06T15:55:47.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-06T15:55:47.269 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-12-06T15:55:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.status 2023-12-06T15:55:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.status 2023-12-06T15:55:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-06T15:55:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=163 2023-12-06T15:55:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 163 = 163 2023-12-06T15:55:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-12-06T15:55:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:55:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-12-06T15:55:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_23': 2023-12-06T15:55:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:55:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:55:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2b100f9ea 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2b100f9ea 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:25 2023 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:25 2023 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:25 2023 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c279a46f-dce9-48d0-be3a-726f061d7181 2023-12-06T15:55:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_23 snap 2023-12-06T15:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T15:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_23.export 2023-12-06T15:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.export 2023-12-06T15:55:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_23.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.export 2023-12-06T15:55:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_23.export 2023-12-06T15:55:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.export 2023-12-06T15:55:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_23.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.export 2023-12-06T15:55:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_23.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_23.export 2023-12-06T15:55:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:55:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_24 2023-12-06T15:55:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_24 snap 2023-12-06T15:55:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-12-06T15:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-12-06T15:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-12-06T15:55:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:55:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:55:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:55:51.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-12-06T15:55:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:55:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:55:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:55:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:55:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:55:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:55:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:55:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:55:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-06T15:55:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:55:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:55:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:55:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:55:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:55:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:55:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-06T15:55:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-06T15:55:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:55:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:55:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-06T15:55:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:55:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:55:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:55:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:55:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:55:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:55:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-06T15:55:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:55:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:55:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:55:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:55:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:55:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_24.status 2023-12-06T15:55:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_24.status 2023-12-06T15:55:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_24.status 2023-12-06T15:55:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_24.status 2023-12-06T15:55:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_24.status 2023-12-06T15:55:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_24.status 2023-12-06T15:55:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-12-06T15:55:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-12-06T15:55:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 165 2023-12-06T15:55:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_24.status 2023-12-06T15:55:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_24.status 2023-12-06T15:55:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-12-06T15:55:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:55:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=165 2023-12-06T15:55:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:55:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_24.status 2023-12-06T15:55:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_24.status 2023-12-06T15:55:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-06T15:55:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=91 2023-12-06T15:55:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 165 = 91 2023-12-06T15:55:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:55:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_24.status 2023-12-06T15:55:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_24.status 2023-12-06T15:55:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-06T15:55:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:55:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:55:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:55:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:55:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 165 = '' 2023-12-06T15:55:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:55:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_24.status 2023-12-06T15:55:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_24.status 2023-12-06T15:55:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-06T15:55:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=165 2023-12-06T15:55:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 165 = 165 2023-12-06T15:55:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-12-06T15:55:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:55:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-12-06T15:55:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_24': 2023-12-06T15:55:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:55:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:55:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:55:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b26f6304ea 2023-12-06T15:55:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b26f6304ea 2023-12-06T15:55:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:55:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:55:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:55:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:55:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:30 2023 2023-12-06T15:55:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:30 2023 2023-12-06T15:55:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:30 2023 2023-12-06T15:55:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:55:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:55:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:55:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: a3bf4bbf-7635-41c7-8700-c2fe78a3fbe0 2023-12-06T15:55:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:55:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_24 snap 2023-12-06T15:55:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T15:55:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:55:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_24.export 2023-12-06T15:55:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_24.export 2023-12-06T15:55:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_24.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_24.export 2023-12-06T15:55:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_24.export 2023-12-06T15:55:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_24.export 2023-12-06T15:55:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:55:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_24.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_24.export 2023-12-06T15:55:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_24.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_24.export 2023-12-06T15:55:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:55:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_25 2023-12-06T15:55:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_25 snap 2023-12-06T15:55:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:55:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:55:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:55:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-12-06T15:55:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:55:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-12-06T15:55:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:55:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:55:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:55:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:55:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-12-06T15:55:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:55:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:55:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:55:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:55:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-12-06T15:55:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:55:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:55:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:55:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:55:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:55:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:55:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:55:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:55:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:55:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-12-06T15:55:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:55:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:55:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:55:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:55:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-12-06T15:55:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-06T15:55:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:55:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:55:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:55:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2023-12-06T15:55:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:55:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:55:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:55:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:55:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:55:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:55:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:55:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:55:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:55:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:55:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:55:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:55:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:55:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:55:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:55:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:55:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:55:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2023-12-06T15:55:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:55:58.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:55:58.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:55:58.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:55:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:55:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:55:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:55:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:55:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-12-06T15:55:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:55:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:55:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:55:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-12-06T15:55:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:55:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:55:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_25.status 2023-12-06T15:55:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_25.status 2023-12-06T15:55:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_25.status 2023-12-06T15:55:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_25.status 2023-12-06T15:55:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_25.status 2023-12-06T15:55:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:55:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.status 2023-12-06T15:55:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-12-06T15:55:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:55:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-12-06T15:55:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 167 2023-12-06T15:55:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_25.status 2023-12-06T15:55:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:55:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:55:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_25.status 2023-12-06T15:55:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-12-06T15:55:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:55:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=167 2023-12-06T15:55:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:55:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:55:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.status 2023-12-06T15:55:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:55:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.status 2023-12-06T15:55:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-06T15:55:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:55:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:55:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=92 2023-12-06T15:55:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 167 = 92 2023-12-06T15:55:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:55:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:55:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.status 2023-12-06T15:55:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:55:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:55:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:55:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.status 2023-12-06T15:55:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-06T15:55:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 167 = '' 2023-12-06T15:56:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:56:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.status 2023-12-06T15:56:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:56:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.status 2023-12-06T15:56:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-06T15:56:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 167 = '' 2023-12-06T15:56:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:56:02.443 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8308525441943261) > weight (0.25) 2023-12-06T15:56:02.443 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8356383082299029) > weight (0.25) 2023-12-06T15:56:02.443 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-06T15:56:02.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-06T15:56:02.443 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-06T15:56:02.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi204.stderr:2023-12-06T15:56:02.442+0000 7f0c25f32700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 125565) UID: 1000 2023-12-06T15:56:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.status 2023-12-06T15:56:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:56:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.status 2023-12-06T15:56:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-06T15:56:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=167 2023-12-06T15:56:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 167 = 167 2023-12-06T15:56:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-12-06T15:56:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:56:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:56:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:02.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:56:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-12-06T15:56:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_25': 2023-12-06T15:56:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:56:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:56:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:56:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2d590e1b8 2023-12-06T15:56:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2d590e1b8 2023-12-06T15:56:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:56:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:56:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:56:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:56:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:41 2023 2023-12-06T15:56:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:41 2023 2023-12-06T15:56:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:41 2023 2023-12-06T15:56:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:56:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:56:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:56:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: db2fb06f-c4e5-4af2-8764-3691b6d24393 2023-12-06T15:56:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:56:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_25 snap 2023-12-06T15:56:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T15:56:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:56:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_25.export 2023-12-06T15:56:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.export 2023-12-06T15:56:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_25.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.export 2023-12-06T15:56:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_25.export 2023-12-06T15:56:03.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.export 2023-12-06T15:56:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_25.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.export 2023-12-06T15:56:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_25.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_25.export 2023-12-06T15:56:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:56:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_26 2023-12-06T15:56:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_26 snap 2023-12-06T15:56:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:56:04.444 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-06T15:56:04.444 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:56:04.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:56:04.444 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-06T15:56:04.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-06T15:56:04.445 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-06T15:56:04.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-06T15:56:04.447 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2023-12-06T15:56:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-12-06T15:56:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-12-06T15:56:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:56:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-12-06T15:56:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:56:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:56:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:56:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2023-12-06T15:56:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:56:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:56:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:56:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:56:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:56:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:56:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:56:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:56:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:56:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:56:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:56:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:56:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:56:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:56:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:56:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-12-06T15:56:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-06T15:56:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:56:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:56:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2023-12-06T15:56:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:56:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:56:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2023-12-06T15:56:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:56:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:56:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:56:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:56:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:56:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:56:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:56:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_26.status 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_26.status 2023-12-06T15:56:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_26.status 2023-12-06T15:56:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_26.status 2023-12-06T15:56:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_26.status 2023-12-06T15:56:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.status 2023-12-06T15:56:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-12-06T15:56:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-12-06T15:56:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 169 2023-12-06T15:56:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_26.status 2023-12-06T15:56:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_26.status 2023-12-06T15:56:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-12-06T15:56:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:56:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=169 2023-12-06T15:56:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.status 2023-12-06T15:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.status 2023-12-06T15:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-06T15:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=93 2023-12-06T15:56:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 169 = 93 2023-12-06T15:56:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:56:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.status 2023-12-06T15:56:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.status 2023-12-06T15:56:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-06T15:56:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 169 = '' 2023-12-06T15:56:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:56:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.status 2023-12-06T15:56:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.status 2023-12-06T15:56:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-06T15:56:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 169 = '' 2023-12-06T15:56:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:56:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.status 2023-12-06T15:56:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.status 2023-12-06T15:56:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-06T15:56:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=169 2023-12-06T15:56:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 169 = 169 2023-12-06T15:56:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-12-06T15:56:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:56:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-12-06T15:56:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_26': 2023-12-06T15:56:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:56:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:56:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b299b6bd9 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b299b6bd9 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:49 2023 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:49 2023 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:49 2023 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 8f93da9b-50b8-4808-bf76-a4022502be07 2023-12-06T15:56:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:56:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_26 snap 2023-12-06T15:56:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T15:56:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:56:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_26.export 2023-12-06T15:56:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.export 2023-12-06T15:56:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_26.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.export 2023-12-06T15:56:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_26.export 2023-12-06T15:56:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.export 2023-12-06T15:56:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_26.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.export 2023-12-06T15:56:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_26.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_26.export 2023-12-06T15:56:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:56:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_27 2023-12-06T15:56:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_27 snap 2023-12-06T15:56:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:56:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-12-06T15:56:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-12-06T15:56:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:56:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-12-06T15:56:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:56:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:56:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:56:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2023-12-06T15:56:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:56:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:56:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:56:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:56:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:56:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:56:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:56:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:56:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:56:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:56:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:56:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:56:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:56:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:56:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:56:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-12-06T15:56:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-12-06T15:56:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:56:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:56:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-12-06T15:56:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:56:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:56:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:56:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:56:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:56:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-12-06T15:56:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:56:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:56:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:56:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:56:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:56:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:56:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_27.status 2023-12-06T15:56:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_27.status 2023-12-06T15:56:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_27.status 2023-12-06T15:56:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_27.status 2023-12-06T15:56:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_27.status 2023-12-06T15:56:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.status 2023-12-06T15:56:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-12-06T15:56:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-12-06T15:56:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 171 2023-12-06T15:56:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_27.status 2023-12-06T15:56:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_27.status 2023-12-06T15:56:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-12-06T15:56:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:56:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=171 2023-12-06T15:56:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:56:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.status 2023-12-06T15:56:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.status 2023-12-06T15:56:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-06T15:56:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=94 2023-12-06T15:56:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 171 = 94 2023-12-06T15:56:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:56:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.status 2023-12-06T15:56:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.status 2023-12-06T15:56:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-06T15:56:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:16.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 171 = '' 2023-12-06T15:56:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:56:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.status 2023-12-06T15:56:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.status 2023-12-06T15:56:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-06T15:56:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 171 = '' 2023-12-06T15:56:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:56:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.status 2023-12-06T15:56:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.status 2023-12-06T15:56:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-06T15:56:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=171 2023-12-06T15:56:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 171 = 171 2023-12-06T15:56:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-12-06T15:56:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:56:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-12-06T15:56:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_27': 2023-12-06T15:56:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:56:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:56:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:56:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2b589f5b2 2023-12-06T15:56:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2b589f5b2 2023-12-06T15:56:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:56:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:56:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:56:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:56:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:47:59 2023 2023-12-06T15:56:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:47:59 2023 2023-12-06T15:56:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:47:59 2023 2023-12-06T15:56:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:56:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:56:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:56:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c0035e14-7c4a-401d-a289-cee1fa7ce091 2023-12-06T15:56:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:56:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_27 snap 2023-12-06T15:56:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T15:56:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:56:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_27.export 2023-12-06T15:56:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.export 2023-12-06T15:56:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_27.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.export 2023-12-06T15:56:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_27.export 2023-12-06T15:56:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.export 2023-12-06T15:56:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:20.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_27.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.export 2023-12-06T15:56:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_27.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_27.export 2023-12-06T15:56:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:56:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_28 2023-12-06T15:56:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_28 snap 2023-12-06T15:56:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:56:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-12-06T15:56:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-12-06T15:56:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:56:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-12-06T15:56:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:56:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:56:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:56:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2023-12-06T15:56:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:56:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:56:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:56:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:56:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:56:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:56:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:56:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:56:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-12-06T15:56:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:56:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:56:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:56:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:56:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:56:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:56:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:56:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-12-06T15:56:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-06T15:56:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:56:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:56:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2023-12-06T15:56:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:56:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:56:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:56:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2023-12-06T15:56:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:56:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:56:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:56:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:56:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:56:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:56:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_28.status 2023-12-06T15:56:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_28.status 2023-12-06T15:56:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_28.status 2023-12-06T15:56:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_28.status 2023-12-06T15:56:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_28.status 2023-12-06T15:56:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.status 2023-12-06T15:56:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-12-06T15:56:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-12-06T15:56:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 173 2023-12-06T15:56:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_28.status 2023-12-06T15:56:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_28.status 2023-12-06T15:56:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-12-06T15:56:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:56:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=173 2023-12-06T15:56:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:56:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.status 2023-12-06T15:56:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.status 2023-12-06T15:56:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-06T15:56:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=95 2023-12-06T15:56:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 173 = 95 2023-12-06T15:56:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:56:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.status 2023-12-06T15:56:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.status 2023-12-06T15:56:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-06T15:56:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 173 = '' 2023-12-06T15:56:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:56:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.status 2023-12-06T15:56:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.status 2023-12-06T15:56:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-06T15:56:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 173 = '' 2023-12-06T15:56:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:56:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.status 2023-12-06T15:56:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.status 2023-12-06T15:56:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-06T15:56:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=173 2023-12-06T15:56:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 173 = 173 2023-12-06T15:56:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-12-06T15:56:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:56:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-12-06T15:56:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_28': 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2f1e3c4ab 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2f1e3c4ab 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:48:08 2023 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:48:08 2023 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:48:08 2023 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:56:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 45a35e04-1f3a-4f6c-a378-30b333fab016 2023-12-06T15:56:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:56:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_28 snap 2023-12-06T15:56:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T15:56:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:56:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_28.export 2023-12-06T15:56:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.export 2023-12-06T15:56:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_28.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.export 2023-12-06T15:56:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_28.export 2023-12-06T15:56:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.export 2023-12-06T15:56:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_28.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.export 2023-12-06T15:56:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_28.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_28.export 2023-12-06T15:56:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:56:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_29 2023-12-06T15:56:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_29 snap 2023-12-06T15:56:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:56:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-12-06T15:56:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-12-06T15:56:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:56:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-12-06T15:56:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-12-06T15:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:56:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:56:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:56:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-12-06T15:56:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:56:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:56:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:56:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:56:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:56:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-12-06T15:56:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-06T15:56:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:56:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:56:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-12-06T15:56:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:56:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:56:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:56:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:56:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:56:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:56:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:56:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-12-06T15:56:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:56:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:56:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:56:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:56:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:56:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:56:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:56:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-12-06T15:56:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_29.status 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_29.status 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_29.status 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_29.status 2023-12-06T15:56:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_29.status 2023-12-06T15:56:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.status 2023-12-06T15:56:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-12-06T15:56:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-12-06T15:56:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 175 2023-12-06T15:56:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_29.status 2023-12-06T15:56:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_29.status 2023-12-06T15:56:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-12-06T15:56:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:56:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=175 2023-12-06T15:56:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:56:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.status 2023-12-06T15:56:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.status 2023-12-06T15:56:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-06T15:56:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=96 2023-12-06T15:56:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 175 = 96 2023-12-06T15:56:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:56:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.status 2023-12-06T15:56:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.status 2023-12-06T15:56:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-06T15:56:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 175 = '' 2023-12-06T15:56:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:56:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.status 2023-12-06T15:56:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.status 2023-12-06T15:56:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-06T15:56:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 175 = '' 2023-12-06T15:56:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:56:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.status 2023-12-06T15:56:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.status 2023-12-06T15:56:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-06T15:56:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=175 2023-12-06T15:56:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 175 = 175 2023-12-06T15:56:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-12-06T15:56:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:56:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-12-06T15:56:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_29': 2023-12-06T15:56:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:56:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b21060f888 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b21060f888 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:48:16 2023 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:48:16 2023 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:48:16 2023 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 0031e62d-baea-4f3c-837b-04d500983bed 2023-12-06T15:56:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:56:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_29 snap 2023-12-06T15:56:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T15:56:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:56:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_29.export 2023-12-06T15:56:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.export 2023-12-06T15:56:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_29.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.export 2023-12-06T15:56:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_29.export 2023-12-06T15:56:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.export 2023-12-06T15:56:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_29.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.export 2023-12-06T15:56:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_29.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_29.export 2023-12-06T15:56:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:56:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_30 2023-12-06T15:56:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_30 snap 2023-12-06T15:56:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:56:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-12-06T15:56:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-12-06T15:56:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:56:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-12-06T15:56:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:56:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:56:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:56:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2023-12-06T15:56:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:56:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:56:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:56:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:56:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:56:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:56:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:56:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:56:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-12-06T15:56:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:56:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:56:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:56:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:56:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:56:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:56:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:56:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:56:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-12-06T15:56:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-06T15:56:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:56:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:56:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-12-06T15:56:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:56:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:56:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:56:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:56:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-12-06T15:56:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:56:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:56:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:56:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:56:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:56:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:56:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:56:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-12-06T15:56:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:56:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:56:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-12-06T15:56:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_30.status 2023-12-06T15:56:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_30.status 2023-12-06T15:56:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_30.status 2023-12-06T15:56:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_30.status 2023-12-06T15:56:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_30.status 2023-12-06T15:56:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.status 2023-12-06T15:56:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-12-06T15:56:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-12-06T15:56:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 177 2023-12-06T15:56:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_30.status 2023-12-06T15:56:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_30.status 2023-12-06T15:56:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-12-06T15:56:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:56:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=177 2023-12-06T15:56:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:56:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.status 2023-12-06T15:56:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.status 2023-12-06T15:56:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-06T15:56:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=97 2023-12-06T15:56:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 177 = 97 2023-12-06T15:56:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:56:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.status 2023-12-06T15:56:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.status 2023-12-06T15:56:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-06T15:56:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 177 = '' 2023-12-06T15:56:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:56:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.status 2023-12-06T15:56:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.status 2023-12-06T15:56:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-06T15:56:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 177 = '' 2023-12-06T15:56:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:56:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.status 2023-12-06T15:56:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.status 2023-12-06T15:56:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-06T15:56:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=177 2023-12-06T15:56:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 177 = 177 2023-12-06T15:56:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-12-06T15:56:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:56:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-12-06T15:56:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_30': 2023-12-06T15:56:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:56:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:56:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:56:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b277815422 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b277815422 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:48:26 2023 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:48:26 2023 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:48:26 2023 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 393bd03f-74d9-4863-9753-476b06c160ad 2023-12-06T15:56:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:56:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_30 snap 2023-12-06T15:56:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T15:56:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:56:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_30.export 2023-12-06T15:56:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.export 2023-12-06T15:56:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_30.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.export 2023-12-06T15:56:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_30.export 2023-12-06T15:56:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.export 2023-12-06T15:56:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_30.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.export 2023-12-06T15:56:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_30.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_30.export 2023-12-06T15:56:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:56:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_31 2023-12-06T15:56:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_31 snap 2023-12-06T15:56:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:56:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-12-06T15:56:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-12-06T15:56:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:56:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-12-06T15:56:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:56:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:56:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:56:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-12-06T15:56:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:56:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:56:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:56:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:56:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:56:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:56:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:56:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:56:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:56:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:56:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:56:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:56:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:56:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:56:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:56:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:56:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-12-06T15:56:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-06T15:56:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:56:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:56:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2023-12-06T15:56:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:56:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:56:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:56:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:56:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:56:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:56:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:56:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:56:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2023-12-06T15:56:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:56:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:56:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:56:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:56:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:56:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:56:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:56:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-12-06T15:56:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:56:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:56:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-12-06T15:56:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_31.status 2023-12-06T15:56:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_31.status 2023-12-06T15:56:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_31.status 2023-12-06T15:56:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_31.status 2023-12-06T15:56:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_31.status 2023-12-06T15:56:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.status 2023-12-06T15:56:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-12-06T15:56:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-12-06T15:56:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 179 2023-12-06T15:56:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_31.status 2023-12-06T15:56:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_31.status 2023-12-06T15:56:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-12-06T15:56:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:56:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=179 2023-12-06T15:56:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:56:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.status 2023-12-06T15:56:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.status 2023-12-06T15:56:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-06T15:56:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=98 2023-12-06T15:56:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 179 = 98 2023-12-06T15:56:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:56:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.status 2023-12-06T15:56:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.status 2023-12-06T15:56:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-06T15:56:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 179 = '' 2023-12-06T15:56:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:56:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.status 2023-12-06T15:56:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.status 2023-12-06T15:56:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-06T15:56:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 179 = '' 2023-12-06T15:56:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:56:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.status 2023-12-06T15:56:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.status 2023-12-06T15:56:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-06T15:56:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=179 2023-12-06T15:56:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 179 = 179 2023-12-06T15:56:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-12-06T15:56:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:56:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-12-06T15:56:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_31': 2023-12-06T15:56:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:56:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:56:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:56:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2abc0a349 2023-12-06T15:56:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2abc0a349 2023-12-06T15:56:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:56:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:56:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:56:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:56:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:48:39 2023 2023-12-06T15:56:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:48:39 2023 2023-12-06T15:56:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:48:39 2023 2023-12-06T15:56:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:56:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:56:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:56:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c5f2021b-abb5-4a19-80cd-cedca15b14e1 2023-12-06T15:56:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:56:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_31 snap 2023-12-06T15:56:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T15:56:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:56:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_31.export 2023-12-06T15:56:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.export 2023-12-06T15:56:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_31.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.export 2023-12-06T15:56:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_31.export 2023-12-06T15:56:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.export 2023-12-06T15:56:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:56:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_31.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.export 2023-12-06T15:56:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_31.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_31.export 2023-12-06T15:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_32 2023-12-06T15:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_32 snap 2023-12-06T15:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:56:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-12-06T15:56:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-12-06T15:56:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:56:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:56:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-12-06T15:56:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:56:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:56:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:56:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2023-12-06T15:56:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:56:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:56:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:56:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:56:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:56:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:56:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:56:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:56:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:56:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:56:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:56:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:56:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:56:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:56:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:56:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:56:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:56:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:56:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:56:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-12-06T15:56:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-12-06T15:56:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:56:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:56:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:56:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-12-06T15:56:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:56:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:56:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:56:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:56:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:56:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:56:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:56:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:56:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:56:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:56:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:56:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:56:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:56:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:56:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:56:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:56:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-12-06T15:56:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:56:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:56:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:56:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:56:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:56:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_32.status 2023-12-06T15:56:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_32.status 2023-12-06T15:56:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_32.status 2023-12-06T15:56:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_32.status 2023-12-06T15:56:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_32.status 2023-12-06T15:56:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:56:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.status 2023-12-06T15:56:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-12-06T15:56:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-12-06T15:56:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 181 2023-12-06T15:56:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_32.status 2023-12-06T15:56:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:56:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_32.status 2023-12-06T15:56:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-12-06T15:56:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:56:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=181 2023-12-06T15:56:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:56:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.status 2023-12-06T15:56:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.status 2023-12-06T15:56:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-06T15:56:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=99 2023-12-06T15:56:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 181 = 99 2023-12-06T15:56:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:56:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.status 2023-12-06T15:56:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.status 2023-12-06T15:56:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-06T15:56:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 181 = '' 2023-12-06T15:56:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:56:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.status 2023-12-06T15:56:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.status 2023-12-06T15:56:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-06T15:56:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:56:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:56:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:56:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:56:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:56:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 181 = '' 2023-12-06T15:56:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:56:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:56:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.status 2023-12-06T15:56:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.status 2023-12-06T15:56:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-06T15:56:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:56:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:56:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=181 2023-12-06T15:56:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 181 = 181 2023-12-06T15:56:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:56:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-12-06T15:56:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:56:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:56:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:56:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:56:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:56:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:56:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:00.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-12-06T15:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_32': 2023-12-06T15:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b27def284f 2023-12-06T15:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b27def284f 2023-12-06T15:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:48:48 2023 2023-12-06T15:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:48:48 2023 2023-12-06T15:57:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:48:48 2023 2023-12-06T15:57:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:57:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:57:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:57:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 994706d1-52be-4379-8f1d-c660ef60435c 2023-12-06T15:57:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_32 snap 2023-12-06T15:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T15:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_32.export 2023-12-06T15:57:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.export 2023-12-06T15:57:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_32.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.export 2023-12-06T15:57:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_32.export 2023-12-06T15:57:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.export 2023-12-06T15:57:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_32.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.export 2023-12-06T15:57:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_32.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_32.export 2023-12-06T15:57:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:57:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_33 2023-12-06T15:57:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_33 snap 2023-12-06T15:57:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:57:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-12-06T15:57:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-12-06T15:57:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:57:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-12-06T15:57:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:57:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:57:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:57:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2023-12-06T15:57:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:57:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:57:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:57:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:57:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:57:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:57:03.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:57:03.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:57:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-12-06T15:57:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:57:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:57:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:57:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:57:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:57:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:57:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:57:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:57:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:57:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:57:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:57:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:57:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-12-06T15:57:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-06T15:57:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:57:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:57:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-12-06T15:57:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:57:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:57:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:57:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:57:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:57:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:57:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:57:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:57:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-12-06T15:57:03.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:57:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:57:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:57:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:57:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-12-06T15:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:57:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:57:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-12-06T15:57:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_33.status 2023-12-06T15:57:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_33.status 2023-12-06T15:57:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_33.status 2023-12-06T15:57:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_33.status 2023-12-06T15:57:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_33.status 2023-12-06T15:57:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.status 2023-12-06T15:57:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-12-06T15:57:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-12-06T15:57:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 183 2023-12-06T15:57:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_33.status 2023-12-06T15:57:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_33.status 2023-12-06T15:57:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-12-06T15:57:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:57:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=183 2023-12-06T15:57:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:57:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.status 2023-12-06T15:57:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.status 2023-12-06T15:57:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-06T15:57:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=100 2023-12-06T15:57:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 183 = 100 2023-12-06T15:57:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:57:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.status 2023-12-06T15:57:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.status 2023-12-06T15:57:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-06T15:57:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 183 = '' 2023-12-06T15:57:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:57:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.status 2023-12-06T15:57:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.status 2023-12-06T15:57:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-06T15:57:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 183 = '' 2023-12-06T15:57:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:57:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.status 2023-12-06T15:57:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.status 2023-12-06T15:57:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-06T15:57:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=183 2023-12-06T15:57:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 183 = 183 2023-12-06T15:57:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-12-06T15:57:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:57:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-12-06T15:57:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_33': 2023-12-06T15:57:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:57:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:57:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:57:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b281d0c87 2023-12-06T15:57:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b281d0c87 2023-12-06T15:57:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:57:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:57:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:57:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:57:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:49:02 2023 2023-12-06T15:57:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:49:02 2023 2023-12-06T15:57:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:49:02 2023 2023-12-06T15:57:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:57:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:57:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:57:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 3293bbff-63fc-47b0-b7af-aa270d848ce2 2023-12-06T15:57:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:57:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_33 snap 2023-12-06T15:57:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T15:57:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:57:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_33.export 2023-12-06T15:57:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.export 2023-12-06T15:57:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_33.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.export 2023-12-06T15:57:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_33.export 2023-12-06T15:57:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.export 2023-12-06T15:57:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_33.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.export 2023-12-06T15:57:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_33.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_33.export 2023-12-06T15:57:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:57:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_34 2023-12-06T15:57:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_34 snap 2023-12-06T15:57:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:57:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-12-06T15:57:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-12-06T15:57:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:57:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-12-06T15:57:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:57:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:57:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:57:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-12-06T15:57:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:57:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:57:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:57:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:57:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:57:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:57:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:57:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:57:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-06T15:57:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:57:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:57:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:57:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:57:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:57:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:57:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:57:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-06T15:57:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-06T15:57:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:57:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:57:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-06T15:57:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:57:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:57:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:57:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:57:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:57:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:57:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:57:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:57:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-06T15:57:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:57:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:57:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:57:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:57:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:57:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:57:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:57:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-12-06T15:57:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:57:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:57:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-12-06T15:57:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_34.status 2023-12-06T15:57:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_34.status 2023-12-06T15:57:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_34.status 2023-12-06T15:57:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_34.status 2023-12-06T15:57:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_34.status 2023-12-06T15:57:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.status 2023-12-06T15:57:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-12-06T15:57:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-12-06T15:57:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 185 2023-12-06T15:57:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_34.status 2023-12-06T15:57:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_34.status 2023-12-06T15:57:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-12-06T15:57:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:57:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=185 2023-12-06T15:57:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:57:13.448 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9262932060573114) > weight (0.25) 2023-12-06T15:57:13.448 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.581877885769389) > weight (0.25) 2023-12-06T15:57:13.449 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5032424552650726) > weight (0.25) 2023-12-06T15:57:13.449 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4968075471346265) > weight (0.25) 2023-12-06T15:57:13.449 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-12-06T15:57:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.status 2023-12-06T15:57:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.status 2023-12-06T15:57:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-06T15:57:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=101 2023-12-06T15:57:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 185 = 101 2023-12-06T15:57:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:57:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.status 2023-12-06T15:57:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.status 2023-12-06T15:57:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-06T15:57:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 185 = '' 2023-12-06T15:57:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:57:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.status 2023-12-06T15:57:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.status 2023-12-06T15:57:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-06T15:57:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 185 = '' 2023-12-06T15:57:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:57:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.status 2023-12-06T15:57:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.status 2023-12-06T15:57:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-06T15:57:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=185 2023-12-06T15:57:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 185 = 185 2023-12-06T15:57:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-12-06T15:57:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:57:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-12-06T15:57:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_34': 2023-12-06T15:57:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:57:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:57:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:57:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2a0111c25 2023-12-06T15:57:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2a0111c25 2023-12-06T15:57:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:57:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:57:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:57:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:57:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:49:09 2023 2023-12-06T15:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:49:09 2023 2023-12-06T15:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:49:09 2023 2023-12-06T15:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: dc0720e9-3db1-4118-8a6c-3ea4239b21d7 2023-12-06T15:57:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:57:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_34 snap 2023-12-06T15:57:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T15:57:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:57:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_34.export 2023-12-06T15:57:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.export 2023-12-06T15:57:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_34.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.export 2023-12-06T15:57:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_34.export 2023-12-06T15:57:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.export 2023-12-06T15:57:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_34.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.export 2023-12-06T15:57:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_34.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_34.export 2023-12-06T15:57:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:57:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_35 2023-12-06T15:57:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_35 snap 2023-12-06T15:57:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:57:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-12-06T15:57:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-12-06T15:57:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:57:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-12-06T15:57:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:57:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:57:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:57:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2023-12-06T15:57:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:57:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:57:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:57:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:57:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:57:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:57:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:57:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:57:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:57:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:57:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:57:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:57:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:57:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:57:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:57:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-12-06T15:57:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-12-06T15:57:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:57:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:57:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2023-12-06T15:57:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:57:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:57:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:57:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:57:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:57:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:57:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2023-12-06T15:57:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:57:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:57:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:57:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:57:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:57:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:57:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:57:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-12-06T15:57:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:57:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:57:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-12-06T15:57:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_35.status 2023-12-06T15:57:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_35.status 2023-12-06T15:57:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_35.status 2023-12-06T15:57:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_35.status 2023-12-06T15:57:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_35.status 2023-12-06T15:57:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.status 2023-12-06T15:57:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-12-06T15:57:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-12-06T15:57:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 187 2023-12-06T15:57:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_35.status 2023-12-06T15:57:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_35.status 2023-12-06T15:57:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-12-06T15:57:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:57:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=187 2023-12-06T15:57:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:57:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.status 2023-12-06T15:57:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.status 2023-12-06T15:57:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-06T15:57:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=102 2023-12-06T15:57:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 187 = 102 2023-12-06T15:57:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:57:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.status 2023-12-06T15:57:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.status 2023-12-06T15:57:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-06T15:57:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 187 = '' 2023-12-06T15:57:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:57:22.269 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-06T15:57:22.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-06T15:57:22.270 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-06T15:57:22.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T15:57:22.269+0000 7f0120384700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 140644) UID: 1000 2023-12-06T15:57:22.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:57:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.status 2023-12-06T15:57:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.status 2023-12-06T15:57:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-06T15:57:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 187 = '' 2023-12-06T15:57:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:57:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.status 2023-12-06T15:57:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.status 2023-12-06T15:57:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-06T15:57:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=187 2023-12-06T15:57:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 187 = 187 2023-12-06T15:57:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-12-06T15:57:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:57:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:25.271 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-06T15:57:25.271 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:57:25.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:57:25.272 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-06T15:57:25.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-06T15:57:25.272 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-06T15:57:25.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-06T15:57:25.275 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2023-12-06T15:57:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-12-06T15:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_35': 2023-12-06T15:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2b0949ee3 2023-12-06T15:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2b0949ee3 2023-12-06T15:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:49:21 2023 2023-12-06T15:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:49:21 2023 2023-12-06T15:57:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:49:21 2023 2023-12-06T15:57:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:57:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:57:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:57:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 12508ac3-20ec-4ba0-98c2-40a26628f314 2023-12-06T15:57:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:57:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_35 snap 2023-12-06T15:57:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T15:57:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:57:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_35.export 2023-12-06T15:57:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.export 2023-12-06T15:57:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_35.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.export 2023-12-06T15:57:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_35.export 2023-12-06T15:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.export 2023-12-06T15:57:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:25.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_35.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.export 2023-12-06T15:57:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_35.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_35.export 2023-12-06T15:57:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:57:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_36 2023-12-06T15:57:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_36 snap 2023-12-06T15:57:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:57:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-12-06T15:57:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-12-06T15:57:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:57:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-12-06T15:57:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:57:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:57:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:57:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2023-12-06T15:57:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:57:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:57:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:57:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:57:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:57:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:57:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:57:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:57:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:57:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:57:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:57:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:57:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:57:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:57:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:57:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:57:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-12-06T15:57:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-06T15:57:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:57:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:57:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:57:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:57:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:57:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2023-12-06T15:57:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:57:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:57:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:57:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:57:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_36.status 2023-12-06T15:57:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_36.status 2023-12-06T15:57:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_36.status 2023-12-06T15:57:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_36.status 2023-12-06T15:57:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_36.status 2023-12-06T15:57:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.status 2023-12-06T15:57:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-12-06T15:57:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-12-06T15:57:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 189 2023-12-06T15:57:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_36.status 2023-12-06T15:57:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_36.status 2023-12-06T15:57:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-12-06T15:57:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:57:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=189 2023-12-06T15:57:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:57:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.status 2023-12-06T15:57:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.status 2023-12-06T15:57:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-06T15:57:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=103 2023-12-06T15:57:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 189 = 103 2023-12-06T15:57:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:57:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.status 2023-12-06T15:57:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.status 2023-12-06T15:57:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-06T15:57:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 189 = '' 2023-12-06T15:57:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:57:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.status 2023-12-06T15:57:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.status 2023-12-06T15:57:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-06T15:57:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 189 = '' 2023-12-06T15:57:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:57:30.971 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.968+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8dc7cfd40 2023-12-06T15:57:30.973 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8dce084e0 2023-12-06T15:57:30.973 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8dd228000 2023-12-06T15:57:30.973 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8e9c00a90 2023-12-06T15:57:30.973 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8e9c01a00 2023-12-06T15:57:30.973 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8ea2c4750 2023-12-06T15:57:30.973 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8ea6bdc70 2023-12-06T15:57:30.974 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8ea78e5b0 2023-12-06T15:57:30.974 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8ea78eea0 2023-12-06T15:57:30.974 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8eb865ad0 2023-12-06T15:57:30.974 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8eb9be820 2023-12-06T15:57:30.974 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8ed784dd0 2023-12-06T15:57:30.974 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e90a731520 2023-12-06T15:57:30.974 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T15:57:30.971+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e90e18fee0 2023-12-06T15:57:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.status 2023-12-06T15:57:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.status 2023-12-06T15:57:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-06T15:57:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=189 2023-12-06T15:57:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 189 = 189 2023-12-06T15:57:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-12-06T15:57:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:57:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-12-06T15:57:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_36': 2023-12-06T15:57:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:57:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:57:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:57:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2416e858d 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2416e858d 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:49:27 2023 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:49:27 2023 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:49:27 2023 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 4ef4b927-2e3c-4c53-87cf-79905d0ea25b 2023-12-06T15:57:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:57:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_36 snap 2023-12-06T15:57:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T15:57:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:57:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_36.export 2023-12-06T15:57:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.export 2023-12-06T15:57:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_36.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.export 2023-12-06T15:57:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_36.export 2023-12-06T15:57:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.export 2023-12-06T15:57:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_36.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.export 2023-12-06T15:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_36.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_36.export 2023-12-06T15:57:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:57:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_37 2023-12-06T15:57:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_37 snap 2023-12-06T15:57:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:57:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-12-06T15:57:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-12-06T15:57:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:57:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-12-06T15:57:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:57:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:57:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:57:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2023-12-06T15:57:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:57:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:57:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:57:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:57:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:57:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:57:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:57:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:57:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:57:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:57:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:57:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:57:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:57:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-12-06T15:57:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-06T15:57:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:57:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:57:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2023-12-06T15:57:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:57:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:57:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:57:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2023-12-06T15:57:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:57:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:57:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:57:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:57:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:57:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:57:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:57:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-12-06T15:57:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:57:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:57:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-12-06T15:57:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_37.status 2023-12-06T15:57:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_37.status 2023-12-06T15:57:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_37.status 2023-12-06T15:57:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_37.status 2023-12-06T15:57:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_37.status 2023-12-06T15:57:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.status 2023-12-06T15:57:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-12-06T15:57:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-12-06T15:57:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 191 2023-12-06T15:57:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_37.status 2023-12-06T15:57:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_37.status 2023-12-06T15:57:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-12-06T15:57:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:57:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=191 2023-12-06T15:57:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:57:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.status 2023-12-06T15:57:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.status 2023-12-06T15:57:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-06T15:57:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=104 2023-12-06T15:57:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 191 = 104 2023-12-06T15:57:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:57:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.status 2023-12-06T15:57:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.status 2023-12-06T15:57:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-06T15:57:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 191 = '' 2023-12-06T15:57:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:57:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.status 2023-12-06T15:57:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.status 2023-12-06T15:57:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-06T15:57:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 191 = '' 2023-12-06T15:57:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:57:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.status 2023-12-06T15:57:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.status 2023-12-06T15:57:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-06T15:57:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=191 2023-12-06T15:57:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 191 = 191 2023-12-06T15:57:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-12-06T15:57:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:57:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-12-06T15:57:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_37': 2023-12-06T15:57:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:57:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:57:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b252f16b84 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b252f16b84 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:49:40 2023 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:49:40 2023 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:49:40 2023 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 9a78f30b-f892-4362-84e4-ebdc27892076 2023-12-06T15:57:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:57:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_37 snap 2023-12-06T15:57:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T15:57:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:57:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_37.export 2023-12-06T15:57:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.export 2023-12-06T15:57:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_37.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.export 2023-12-06T15:57:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_37.export 2023-12-06T15:57:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.export 2023-12-06T15:57:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_37.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.export 2023-12-06T15:57:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_37.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_37.export 2023-12-06T15:57:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:57:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_38 2023-12-06T15:57:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_38 snap 2023-12-06T15:57:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:57:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2023-12-06T15:57:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2023-12-06T15:57:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:57:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2023-12-06T15:57:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:57:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:57:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:57:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2023-12-06T15:57:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:57:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:57:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:57:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:57:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:57:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:57:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:57:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:57:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-12-06T15:57:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:57:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:57:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:57:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:57:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:57:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:57:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:57:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:57:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:57:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:57:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:57:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:57:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-12-06T15:57:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-12-06T15:57:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:57:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:57:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-12-06T15:57:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:57:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:57:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:57:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:57:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:57:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:57:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-12-06T15:57:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:57:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:57:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:57:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:57:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:57:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:57:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_38.status 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_38.status 2023-12-06T15:57:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_38.status 2023-12-06T15:57:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_38.status 2023-12-06T15:57:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_38.status 2023-12-06T15:57:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.status 2023-12-06T15:57:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2023-12-06T15:57:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2023-12-06T15:57:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 193 2023-12-06T15:57:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_38.status 2023-12-06T15:57:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_38.status 2023-12-06T15:57:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2023-12-06T15:57:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:57:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=193 2023-12-06T15:57:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:57:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.status 2023-12-06T15:57:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.status 2023-12-06T15:57:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-06T15:57:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=105 2023-12-06T15:57:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 193 = 105 2023-12-06T15:57:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:57:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.status 2023-12-06T15:57:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.status 2023-12-06T15:57:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-06T15:57:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 193 = '' 2023-12-06T15:57:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.status 2023-12-06T15:57:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.status 2023-12-06T15:57:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-06T15:57:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 193 = '' 2023-12-06T15:57:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:57:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.status 2023-12-06T15:57:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.status 2023-12-06T15:57:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-06T15:57:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:49.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=193 2023-12-06T15:57:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 193 = 193 2023-12-06T15:57:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2023-12-06T15:57:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:57:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2023-12-06T15:57:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_38': 2023-12-06T15:57:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:57:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:57:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:57:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b21eba8f94 2023-12-06T15:57:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b21eba8f94 2023-12-06T15:57:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:57:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:57:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:57:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:57:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:49:54 2023 2023-12-06T15:57:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:49:54 2023 2023-12-06T15:57:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:49:54 2023 2023-12-06T15:57:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:57:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:57:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:57:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 3f1f4f6b-ba96-4370-b170-e42f93596565 2023-12-06T15:57:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:57:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_38 snap 2023-12-06T15:57:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T15:57:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:57:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_38.export 2023-12-06T15:57:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.export 2023-12-06T15:57:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_38.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.export 2023-12-06T15:57:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_38.export 2023-12-06T15:57:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.export 2023-12-06T15:57:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_38.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.export 2023-12-06T15:57:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_38.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_38.export 2023-12-06T15:57:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:57:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_39 2023-12-06T15:57:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_39 snap 2023-12-06T15:57:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2023-12-06T15:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2023-12-06T15:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2023-12-06T15:57:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:57:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:57:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:57:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2023-12-06T15:57:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:57:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:57:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:57:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:57:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:57:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:57:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:57:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:57:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:57:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-12-06T15:57:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:57:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:57:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:57:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:57:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:57:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-12-06T15:57:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-06T15:57:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:57:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:57:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:57:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:57:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:57:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:57:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:57:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:57:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2023-12-06T15:57:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:57:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:57:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:57:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:57:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:57:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:57:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:57:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2023-12-06T15:57:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:57:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:57:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2023-12-06T15:57:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:57:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_39.status 2023-12-06T15:57:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_39.status 2023-12-06T15:57:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_39.status 2023-12-06T15:57:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_39.status 2023-12-06T15:57:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_39.status 2023-12-06T15:57:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:57:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.status 2023-12-06T15:57:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2023-12-06T15:57:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2023-12-06T15:57:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 195 2023-12-06T15:57:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_39.status 2023-12-06T15:57:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_39.status 2023-12-06T15:57:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2023-12-06T15:57:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:57:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=195 2023-12-06T15:57:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:57:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.status 2023-12-06T15:57:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.status 2023-12-06T15:57:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-06T15:57:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=106 2023-12-06T15:57:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 195 = 106 2023-12-06T15:57:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:57:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.status 2023-12-06T15:57:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.status 2023-12-06T15:57:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-06T15:57:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 195 = '' 2023-12-06T15:57:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:57:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.status 2023-12-06T15:57:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.status 2023-12-06T15:57:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-06T15:57:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:57:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:57:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:57:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:57:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:57:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 195 = '' 2023-12-06T15:57:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:57:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:57:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.status 2023-12-06T15:57:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.status 2023-12-06T15:57:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-06T15:57:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:57:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:57:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=195 2023-12-06T15:57:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 195 = 195 2023-12-06T15:57:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2023-12-06T15:57:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:57:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:57:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:57:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:57:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2023-12-06T15:57:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_39': 2023-12-06T15:57:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:57:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:57:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:57:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2b2a7f334 2023-12-06T15:57:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2b2a7f334 2023-12-06T15:57:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:57:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:57:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:57:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:57:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:50:02 2023 2023-12-06T15:57:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:50:02 2023 2023-12-06T15:57:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:50:02 2023 2023-12-06T15:57:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:57:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:57:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:57:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 70cf571a-cab9-45c9-bf8a-20c249f7dcf1 2023-12-06T15:57:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:57:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_39 snap 2023-12-06T15:57:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T15:57:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:57:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_39.export 2023-12-06T15:57:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.export 2023-12-06T15:57:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_39.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.export 2023-12-06T15:57:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_39.export 2023-12-06T15:57:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.export 2023-12-06T15:57:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:57:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_39.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.export 2023-12-06T15:57:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_39.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_39.export 2023-12-06T15:57:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:57:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:57:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_40 2023-12-06T15:57:59.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_40 snap 2023-12-06T15:57:59.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:57:59.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:57:59.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:57:59.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:57:59.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:58:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2023-12-06T15:58:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2023-12-06T15:58:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:58:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2023-12-06T15:58:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:58:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:58:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2023-12-06T15:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:58:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:58:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:58:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:58:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:58:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:58:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:58:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:58:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:58:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:58:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-12-06T15:58:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-12-06T15:58:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:58:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:58:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-12-06T15:58:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:58:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:58:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:58:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:58:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:58:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:58:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:58:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-12-06T15:58:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:58:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:58:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:58:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:58:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:58:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:58:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:58:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2023-12-06T15:58:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:58:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:58:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2023-12-06T15:58:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_40.status 2023-12-06T15:58:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_40.status 2023-12-06T15:58:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_40.status 2023-12-06T15:58:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_40.status 2023-12-06T15:58:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_40.status 2023-12-06T15:58:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.status 2023-12-06T15:58:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2023-12-06T15:58:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2023-12-06T15:58:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 197 2023-12-06T15:58:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_40.status 2023-12-06T15:58:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_40.status 2023-12-06T15:58:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2023-12-06T15:58:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=197 2023-12-06T15:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.status 2023-12-06T15:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.status 2023-12-06T15:58:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-06T15:58:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=107 2023-12-06T15:58:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 197 = 107 2023-12-06T15:58:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:58:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.status 2023-12-06T15:58:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.status 2023-12-06T15:58:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-06T15:58:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 197 = '' 2023-12-06T15:58:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:58:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.status 2023-12-06T15:58:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.status 2023-12-06T15:58:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-06T15:58:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 197 = '' 2023-12-06T15:58:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:58:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.status 2023-12-06T15:58:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.status 2023-12-06T15:58:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-06T15:58:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=197 2023-12-06T15:58:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 197 = 197 2023-12-06T15:58:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2023-12-06T15:58:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:58:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2023-12-06T15:58:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_40': 2023-12-06T15:58:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b234efcf45 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b234efcf45 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:50:13 2023 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:50:13 2023 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:50:13 2023 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:58:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: bcd56cda-90d7-45de-b6e1-3c9183852338 2023-12-06T15:58:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_40 snap 2023-12-06T15:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T15:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_40.export 2023-12-06T15:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.export 2023-12-06T15:58:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_40.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.export 2023-12-06T15:58:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_40.export 2023-12-06T15:58:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.export 2023-12-06T15:58:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_40.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.export 2023-12-06T15:58:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_40.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_40.export 2023-12-06T15:58:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:58:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_41 2023-12-06T15:58:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_41 snap 2023-12-06T15:58:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:58:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2023-12-06T15:58:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2023-12-06T15:58:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:58:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2023-12-06T15:58:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:58:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:58:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:58:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2023-12-06T15:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:58:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2023-12-06T15:58:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:58:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:58:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:58:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:58:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:58:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:58:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:58:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:58:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2023-12-06T15:58:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-12-06T15:58:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:58:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:58:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2023-12-06T15:58:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:58:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:58:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:58:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:58:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:58:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2023-12-06T15:58:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:58:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:58:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:58:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:58:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_41.status 2023-12-06T15:58:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_41.status 2023-12-06T15:58:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_41.status 2023-12-06T15:58:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_41.status 2023-12-06T15:58:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_41.status 2023-12-06T15:58:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.status 2023-12-06T15:58:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2023-12-06T15:58:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2023-12-06T15:58:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 199 2023-12-06T15:58:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_41.status 2023-12-06T15:58:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_41.status 2023-12-06T15:58:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2023-12-06T15:58:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:58:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=199 2023-12-06T15:58:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:58:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.status 2023-12-06T15:58:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.status 2023-12-06T15:58:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-06T15:58:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=108 2023-12-06T15:58:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 199 = 108 2023-12-06T15:58:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:58:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.status 2023-12-06T15:58:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.status 2023-12-06T15:58:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-06T15:58:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 199 = '' 2023-12-06T15:58:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:58:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.status 2023-12-06T15:58:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.status 2023-12-06T15:58:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-06T15:58:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 199 = '' 2023-12-06T15:58:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:58:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.status 2023-12-06T15:58:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.status 2023-12-06T15:58:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-06T15:58:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=199 2023-12-06T15:58:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 199 = 199 2023-12-06T15:58:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2023-12-06T15:58:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:58:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2023-12-06T15:58:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_41': 2023-12-06T15:58:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:58:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:58:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:58:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2b584789f 2023-12-06T15:58:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2b584789f 2023-12-06T15:58:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:58:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:58:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:58:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:58:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:50:20 2023 2023-12-06T15:58:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:50:20 2023 2023-12-06T15:58:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:50:20 2023 2023-12-06T15:58:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:58:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:58:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:58:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: d19eb783-02bd-4211-9062-fbc83db629fb 2023-12-06T15:58:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_41 snap 2023-12-06T15:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T15:58:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:58:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_41.export 2023-12-06T15:58:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.export 2023-12-06T15:58:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_41.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.export 2023-12-06T15:58:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_41.export 2023-12-06T15:58:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.export 2023-12-06T15:58:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_41.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.export 2023-12-06T15:58:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_41.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_41.export 2023-12-06T15:58:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:58:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_42 2023-12-06T15:58:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_42 snap 2023-12-06T15:58:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:58:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2023-12-06T15:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2023-12-06T15:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2023-12-06T15:58:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:58:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:58:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:58:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2023-12-06T15:58:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:58:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:58:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:58:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:58:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:58:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:58:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:58:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:58:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2023-12-06T15:58:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:58:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:58:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:58:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:58:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:58:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:58:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2023-12-06T15:58:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2023-12-06T15:58:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:58:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:58:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2023-12-06T15:58:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:58:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:58:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:58:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2023-12-06T15:58:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:58:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:58:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:58:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:58:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:58:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:58:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:58:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2023-12-06T15:58:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:58:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:58:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2023-12-06T15:58:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_42.status 2023-12-06T15:58:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_42.status 2023-12-06T15:58:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_42.status 2023-12-06T15:58:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_42.status 2023-12-06T15:58:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_42.status 2023-12-06T15:58:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.status 2023-12-06T15:58:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2023-12-06T15:58:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2023-12-06T15:58:19.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 201 2023-12-06T15:58:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_42.status 2023-12-06T15:58:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_42.status 2023-12-06T15:58:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2023-12-06T15:58:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:58:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=201 2023-12-06T15:58:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:58:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.status 2023-12-06T15:58:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.status 2023-12-06T15:58:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-06T15:58:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=109 2023-12-06T15:58:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 201 = 109 2023-12-06T15:58:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:58:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.status 2023-12-06T15:58:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.status 2023-12-06T15:58:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-06T15:58:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 201 = '' 2023-12-06T15:58:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:58:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.status 2023-12-06T15:58:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.status 2023-12-06T15:58:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-06T15:58:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 201 = '' 2023-12-06T15:58:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:58:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.status 2023-12-06T15:58:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.status 2023-12-06T15:58:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-06T15:58:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=201 2023-12-06T15:58:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 201 = 201 2023-12-06T15:58:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2023-12-06T15:58:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:58:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2023-12-06T15:58:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_42': 2023-12-06T15:58:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:58:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:58:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:58:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b290ff1f0b 2023-12-06T15:58:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b290ff1f0b 2023-12-06T15:58:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:58:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:58:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:58:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:50:27 2023 2023-12-06T15:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:50:27 2023 2023-12-06T15:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:50:27 2023 2023-12-06T15:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 4c2635cc-57ee-4102-bf76-ae809079b50d 2023-12-06T15:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:58:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_42 snap 2023-12-06T15:58:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T15:58:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:58:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_42.export 2023-12-06T15:58:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.export 2023-12-06T15:58:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_42.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.export 2023-12-06T15:58:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_42.export 2023-12-06T15:58:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.export 2023-12-06T15:58:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_42.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.export 2023-12-06T15:58:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_42.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_42.export 2023-12-06T15:58:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:58:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_43 2023-12-06T15:58:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_43 snap 2023-12-06T15:58:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:58:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2023-12-06T15:58:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2023-12-06T15:58:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:58:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2023-12-06T15:58:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:58:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:58:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:58:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2023-12-06T15:58:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:58:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:58:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:58:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:58:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:58:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:58:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:58:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:58:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2023-12-06T15:58:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:58:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:58:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:58:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:58:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:58:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:58:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:58:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:58:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:58:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:58:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:58:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:58:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2023-12-06T15:58:26.276 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2895671243197594) > weight (0.25) 2023-12-06T15:58:26.276 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5300240353154209) > weight (0.25) 2023-12-06T15:58:26.276 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8481725236648691) > weight (0.25) 2023-12-06T15:58:26.276 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6919917021389688) > weight (0.25) 2023-12-06T15:58:26.276 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-12-06T15:58:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-12-06T15:58:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:58:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2023-12-06T15:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:58:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:58:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:58:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:58:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2023-12-06T15:58:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:58:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:58:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:58:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:58:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:58:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:58:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:58:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2023-12-06T15:58:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:58:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:58:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2023-12-06T15:58:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_43.status 2023-12-06T15:58:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_43.status 2023-12-06T15:58:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_43.status 2023-12-06T15:58:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_43.status 2023-12-06T15:58:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_43.status 2023-12-06T15:58:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.status 2023-12-06T15:58:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2023-12-06T15:58:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2023-12-06T15:58:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 203 2023-12-06T15:58:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_43.status 2023-12-06T15:58:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_43.status 2023-12-06T15:58:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2023-12-06T15:58:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:58:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=203 2023-12-06T15:58:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:58:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.status 2023-12-06T15:58:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.status 2023-12-06T15:58:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-06T15:58:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=110 2023-12-06T15:58:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 203 = 110 2023-12-06T15:58:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:58:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.status 2023-12-06T15:58:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.status 2023-12-06T15:58:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-06T15:58:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 203 = '' 2023-12-06T15:58:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:58:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.status 2023-12-06T15:58:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.status 2023-12-06T15:58:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-06T15:58:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 203 = '' 2023-12-06T15:58:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:58:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.status 2023-12-06T15:58:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.status 2023-12-06T15:58:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-06T15:58:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=203 2023-12-06T15:58:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 203 = 203 2023-12-06T15:58:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2023-12-06T15:58:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:58:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2023-12-06T15:58:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_43': 2023-12-06T15:58:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:58:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b25d2981e2 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b25d2981e2 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:50:39 2023 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:50:39 2023 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:50:39 2023 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:58:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: b9035ac9-e202-49a4-90ed-1f7ed899a231 2023-12-06T15:58:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:58:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_43 snap 2023-12-06T15:58:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T15:58:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:58:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_43.export 2023-12-06T15:58:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.export 2023-12-06T15:58:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_43.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.export 2023-12-06T15:58:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_43.export 2023-12-06T15:58:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.export 2023-12-06T15:58:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_43.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.export 2023-12-06T15:58:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_43.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_43.export 2023-12-06T15:58:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:58:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_44 2023-12-06T15:58:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_44 snap 2023-12-06T15:58:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:58:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2023-12-06T15:58:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2023-12-06T15:58:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:58:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2023-12-06T15:58:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:58:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:58:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:58:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2023-12-06T15:58:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:58:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:58:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:58:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:58:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:58:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:58:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:58:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:58:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-06T15:58:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:58:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:58:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:58:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:58:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:58:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:58:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:58:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:58:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:58:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-06T15:58:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-06T15:58:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:58:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:58:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-12-06T15:58:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:58:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:58:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:58:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:58:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-12-06T15:58:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:58:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:58:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:58:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:58:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2023-12-06T15:58:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:58:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:58:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2023-12-06T15:58:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_44.status 2023-12-06T15:58:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_44.status 2023-12-06T15:58:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_44.status 2023-12-06T15:58:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_44.status 2023-12-06T15:58:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_44.status 2023-12-06T15:58:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.status 2023-12-06T15:58:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2023-12-06T15:58:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2023-12-06T15:58:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 205 2023-12-06T15:58:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_44.status 2023-12-06T15:58:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_44.status 2023-12-06T15:58:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2023-12-06T15:58:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:58:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=205 2023-12-06T15:58:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:58:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.status 2023-12-06T15:58:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.status 2023-12-06T15:58:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-06T15:58:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=111 2023-12-06T15:58:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 205 = 111 2023-12-06T15:58:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:58:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.status 2023-12-06T15:58:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.status 2023-12-06T15:58:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-06T15:58:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 205 = '' 2023-12-06T15:58:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:58:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.status 2023-12-06T15:58:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.status 2023-12-06T15:58:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-06T15:58:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 205 = '' 2023-12-06T15:58:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:58:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.status 2023-12-06T15:58:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.status 2023-12-06T15:58:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-06T15:58:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:38.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=205 2023-12-06T15:58:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 205 = 205 2023-12-06T15:58:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2023-12-06T15:58:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:58:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2023-12-06T15:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_44': 2023-12-06T15:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:58:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2e7a5d0f5 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2e7a5d0f5 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:50:49 2023 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:50:49 2023 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:50:49 2023 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 9a98f3b9-5d2f-447d-9e2f-0f9938f259fe 2023-12-06T15:58:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:58:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_44 snap 2023-12-06T15:58:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T15:58:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:58:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_44.export 2023-12-06T15:58:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.export 2023-12-06T15:58:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_44.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.export 2023-12-06T15:58:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_44.export 2023-12-06T15:58:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.export 2023-12-06T15:58:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_44.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.export 2023-12-06T15:58:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_44.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_44.export 2023-12-06T15:58:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:58:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_45 2023-12-06T15:58:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_45 snap 2023-12-06T15:58:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:58:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2023-12-06T15:58:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2023-12-06T15:58:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:58:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2023-12-06T15:58:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:58:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:58:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:58:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2023-12-06T15:58:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:58:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:58:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:58:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:58:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:58:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:58:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:58:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:58:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2023-12-06T15:58:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:58:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:58:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:58:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:58:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:58:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:58:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:58:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2023-12-06T15:58:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-06T15:58:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:58:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:58:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:58:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:58:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:58:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:58:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2023-12-06T15:58:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:58:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:58:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:58:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:58:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_45.status 2023-12-06T15:58:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_45.status 2023-12-06T15:58:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_45.status 2023-12-06T15:58:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_45.status 2023-12-06T15:58:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_45.status 2023-12-06T15:58:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.status 2023-12-06T15:58:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2023-12-06T15:58:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2023-12-06T15:58:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 207 2023-12-06T15:58:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_45.status 2023-12-06T15:58:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_45.status 2023-12-06T15:58:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2023-12-06T15:58:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:58:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=207 2023-12-06T15:58:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:58:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.status 2023-12-06T15:58:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.status 2023-12-06T15:58:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-06T15:58:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=112 2023-12-06T15:58:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 207 = 112 2023-12-06T15:58:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:58:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.status 2023-12-06T15:58:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.status 2023-12-06T15:58:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-06T15:58:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 207 = '' 2023-12-06T15:58:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.status 2023-12-06T15:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.status 2023-12-06T15:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-06T15:58:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 207 = '' 2023-12-06T15:58:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:58:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.status 2023-12-06T15:58:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.status 2023-12-06T15:58:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-06T15:58:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=207 2023-12-06T15:58:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 207 = 207 2023-12-06T15:58:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2023-12-06T15:58:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:58:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2023-12-06T15:58:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_45': 2023-12-06T15:58:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:58:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:58:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:58:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2810b7371 2023-12-06T15:58:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2810b7371 2023-12-06T15:58:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:58:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:58:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:58:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:58:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:51:03 2023 2023-12-06T15:58:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:51:03 2023 2023-12-06T15:58:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:51:03 2023 2023-12-06T15:58:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:58:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:58:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:58:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 58132a72-7f69-4342-8240-26753ebd27ff 2023-12-06T15:58:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:58:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_45 snap 2023-12-06T15:58:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T15:58:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:58:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_45.export 2023-12-06T15:58:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.export 2023-12-06T15:58:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_45.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.export 2023-12-06T15:58:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_45.export 2023-12-06T15:58:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.export 2023-12-06T15:58:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_45.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.export 2023-12-06T15:58:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_45.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_45.export 2023-12-06T15:58:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:58:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_46 2023-12-06T15:58:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_46 snap 2023-12-06T15:58:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:58:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2023-12-06T15:58:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2023-12-06T15:58:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:58:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2023-12-06T15:58:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:58:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:58:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:58:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2023-12-06T15:58:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:58:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:58:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:58:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:58:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:58:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:58:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:58:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:58:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2023-12-06T15:58:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:58:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:58:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:58:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:58:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:58:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:58:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:58:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2023-12-06T15:58:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-12-06T15:58:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:58:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2023-12-06T15:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:58:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:58:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:58:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:58:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:58:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:58:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:58:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2023-12-06T15:58:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:58:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:58:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:58:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:58:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:58:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:58:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:58:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2023-12-06T15:58:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:58:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:58:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2023-12-06T15:58:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_46.status 2023-12-06T15:58:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_46.status 2023-12-06T15:58:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_46.status 2023-12-06T15:58:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_46.status 2023-12-06T15:58:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_46.status 2023-12-06T15:58:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.status 2023-12-06T15:58:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2023-12-06T15:58:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2023-12-06T15:58:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 209 2023-12-06T15:58:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_46.status 2023-12-06T15:58:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_46.status 2023-12-06T15:58:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2023-12-06T15:58:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:58:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=209 2023-12-06T15:58:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:58:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.status 2023-12-06T15:58:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.status 2023-12-06T15:58:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-06T15:58:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=113 2023-12-06T15:58:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 209 = 113 2023-12-06T15:58:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:58:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.status 2023-12-06T15:58:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.status 2023-12-06T15:58:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-06T15:58:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 209 = '' 2023-12-06T15:58:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:58:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.status 2023-12-06T15:58:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.status 2023-12-06T15:58:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-06T15:58:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:58:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:58:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:58:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:58:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:58:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 209 = '' 2023-12-06T15:58:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:58:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.status 2023-12-06T15:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.status 2023-12-06T15:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-06T15:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:58:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:58:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=209 2023-12-06T15:58:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 209 = 209 2023-12-06T15:58:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2023-12-06T15:58:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:58:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:55.449 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7506897302848949) > weight (0.25) 2023-12-06T15:58:55.449 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7414835433500776) > weight (0.25) 2023-12-06T15:58:55.450 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3544717509958365) > weight (0.25) 2023-12-06T15:58:55.450 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.730978488270128) > weight (0.25) 2023-12-06T15:58:55.450 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2023-12-06T15:58:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2023-12-06T15:58:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_46': 2023-12-06T15:58:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b24c2dd1ed 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b24c2dd1ed 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:51:15 2023 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:51:15 2023 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:51:15 2023 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:58:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c54283d5-014c-482e-9681-a7b875a0863c 2023-12-06T15:58:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:58:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_46 snap 2023-12-06T15:58:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T15:58:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:58:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_46.export 2023-12-06T15:58:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.export 2023-12-06T15:58:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_46.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.export 2023-12-06T15:58:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_46.export 2023-12-06T15:58:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.export 2023-12-06T15:58:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:58:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_46.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.export 2023-12-06T15:58:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_46.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_46.export 2023-12-06T15:58:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:58:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_47 2023-12-06T15:58:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_47 snap 2023-12-06T15:58:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:58:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:58:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:58:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2023-12-06T15:58:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:58:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2023-12-06T15:58:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:58:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:58:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:58:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:58:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:58:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:58:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:58:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:58:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:58:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:58:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-12-06T15:58:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:58:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:58:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:58:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:58:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:58:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:58:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:58:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:58:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:58:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-12-06T15:58:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-06T15:58:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:58:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:58:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:58:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:58:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:58:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:58:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2023-12-06T15:58:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:58:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:58:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:58:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:58:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:58:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:58:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:58:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:58:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2023-12-06T15:58:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:58:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:58:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:58:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2023-12-06T15:58:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:58:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:58:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_47.status 2023-12-06T15:58:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_47.status 2023-12-06T15:58:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_47.status 2023-12-06T15:58:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_47.status 2023-12-06T15:58:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_47.status 2023-12-06T15:58:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:58:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.status 2023-12-06T15:58:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2023-12-06T15:58:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:58:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:58:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:58:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2023-12-06T15:59:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 211 2023-12-06T15:59:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_47.status 2023-12-06T15:59:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:59:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_47.status 2023-12-06T15:59:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2023-12-06T15:59:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:59:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=211 2023-12-06T15:59:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:59:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:59:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.status 2023-12-06T15:59:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:59:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.status 2023-12-06T15:59:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-06T15:59:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=114 2023-12-06T15:59:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 211 = 114 2023-12-06T15:59:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:59:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.status 2023-12-06T15:59:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:59:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.status 2023-12-06T15:59:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-06T15:59:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:59:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:59:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:59:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:59:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:59:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 211 = '' 2023-12-06T15:59:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:59:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.status 2023-12-06T15:59:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:59:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.status 2023-12-06T15:59:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-06T15:59:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:59:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:59:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:59:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:59:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:59:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 211 = '' 2023-12-06T15:59:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:59:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.status 2023-12-06T15:59:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:59:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.status 2023-12-06T15:59:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-06T15:59:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=211 2023-12-06T15:59:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 211 = 211 2023-12-06T15:59:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2023-12-06T15:59:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:59:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:59:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:59:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:59:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:59:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2023-12-06T15:59:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_47': 2023-12-06T15:59:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:59:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b215b7a93b 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b215b7a93b 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:51:22 2023 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:51:22 2023 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:51:22 2023 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c4567031-4494-4727-b144-9c229376da8f 2023-12-06T15:59:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_47 snap 2023-12-06T15:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T15:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_47.export 2023-12-06T15:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.export 2023-12-06T15:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_47.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.export 2023-12-06T15:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_47.export 2023-12-06T15:59:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:59:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.export 2023-12-06T15:59:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:59:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_47.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.export 2023-12-06T15:59:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_47.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_47.export 2023-12-06T15:59:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_48 2023-12-06T15:59:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_48 snap 2023-12-06T15:59:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:59:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:59:05.607 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:05.603+0000 7f44ae2f8700 -1 reset not still connected to 0x559a21ca5ba0 2023-12-06T15:59:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2023-12-06T15:59:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2023-12-06T15:59:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:59:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:59:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:59:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:59:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2023-12-06T15:59:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:59:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:59:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:59:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2023-12-06T15:59:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:59:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:59:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:59:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:59:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:59:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:59:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:59:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:59:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:59:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:59:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:59:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2023-12-06T15:59:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:59:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:59:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:59:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:59:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:59:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:59:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:59:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:59:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:59:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:59:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:59:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:59:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:59:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:59:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:59:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:59:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:59:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:59:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:59:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:59:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:59:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2023-12-06T15:59:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2023-12-06T15:59:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:59:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:59:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:59:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:59:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:59:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:59:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:59:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:59:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:59:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2023-12-06T15:59:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:59:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:59:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:59:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:59:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:59:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:59:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:59:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2023-12-06T15:59:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:59:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:59:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:59:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2023-12-06T15:59:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:59:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_48.status 2023-12-06T15:59:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_48.status 2023-12-06T15:59:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:59:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_48.status 2023-12-06T15:59:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_48.status 2023-12-06T15:59:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_48.status 2023-12-06T15:59:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:59:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.status 2023-12-06T15:59:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2023-12-06T15:59:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2023-12-06T15:59:08.205 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:08.202+0000 7f44ae2f8700 -1 reset not still connected to 0x559a344dcdd0 2023-12-06T15:59:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 213 2023-12-06T15:59:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_48.status 2023-12-06T15:59:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_48.status 2023-12-06T15:59:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2023-12-06T15:59:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:59:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=213 2023-12-06T15:59:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:59:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:08.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:59:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.status 2023-12-06T15:59:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.status 2023-12-06T15:59:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-06T15:59:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=115 2023-12-06T15:59:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 213 = 115 2023-12-06T15:59:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:59:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.status 2023-12-06T15:59:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.status 2023-12-06T15:59:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-06T15:59:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:59:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:59:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:59:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:59:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:59:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 213 = '' 2023-12-06T15:59:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:59:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.status 2023-12-06T15:59:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.status 2023-12-06T15:59:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-06T15:59:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:59:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:59:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:59:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:59:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:59:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 213 = '' 2023-12-06T15:59:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:59:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.status 2023-12-06T15:59:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.status 2023-12-06T15:59:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-06T15:59:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=213 2023-12-06T15:59:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 213 = 213 2023-12-06T15:59:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2023-12-06T15:59:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:59:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:59:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:59:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:59:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2023-12-06T15:59:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_48': 2023-12-06T15:59:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:59:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:59:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:59:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b2cd3004ac 2023-12-06T15:59:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b2cd3004ac 2023-12-06T15:59:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:59:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:59:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:59:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:59:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:51:35 2023 2023-12-06T15:59:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:51:35 2023 2023-12-06T15:59:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:51:35 2023 2023-12-06T15:59:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:59:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:59:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:59:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 3ec3a26c-39fa-4d23-b6cb-9a34c415483f 2023-12-06T15:59:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:59:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_48 snap 2023-12-06T15:59:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T15:59:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:59:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:59:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_48.export 2023-12-06T15:59:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.export 2023-12-06T15:59:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_48.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.export 2023-12-06T15:59:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_48.export 2023-12-06T15:59:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:59:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.export 2023-12-06T15:59:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:59:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_48.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.export 2023-12-06T15:59:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_48.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_48.export 2023-12-06T15:59:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_49 2023-12-06T15:59:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_49 snap 2023-12-06T15:59:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:59:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:59:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2023-12-06T15:59:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2023-12-06T15:59:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:59:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:59:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:59:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:59:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2023-12-06T15:59:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:59:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:59:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:59:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2023-12-06T15:59:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:59:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:59:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:59:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:59:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:59:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:59:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:59:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:59:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:59:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:59:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:59:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:59:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:59:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:59:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:59:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:59:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:59:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:59:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:59:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2023-12-06T15:59:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-12-06T15:59:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:59:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:59:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:59:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2023-12-06T15:59:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:59:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:59:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:59:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:59:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:59:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:59:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2023-12-06T15:59:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:59:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:59:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:59:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:59:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_49.status 2023-12-06T15:59:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_49.status 2023-12-06T15:59:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:59:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_49.status 2023-12-06T15:59:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_49.status 2023-12-06T15:59:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_49.status 2023-12-06T15:59:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:59:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.status 2023-12-06T15:59:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2023-12-06T15:59:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2023-12-06T15:59:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 215 2023-12-06T15:59:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_49.status 2023-12-06T15:59:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_49.status 2023-12-06T15:59:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2023-12-06T15:59:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:59:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=215 2023-12-06T15:59:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:59:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:59:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.status 2023-12-06T15:59:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.status 2023-12-06T15:59:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-06T15:59:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=116 2023-12-06T15:59:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 215 = 116 2023-12-06T15:59:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:59:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.status 2023-12-06T15:59:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.status 2023-12-06T15:59:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-06T15:59:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:59:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:59:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:59:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:59:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:59:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 215 = '' 2023-12-06T15:59:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:59:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.status 2023-12-06T15:59:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.status 2023-12-06T15:59:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-06T15:59:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:59:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:59:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:59:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:59:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:59:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 215 = '' 2023-12-06T15:59:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:59:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.status 2023-12-06T15:59:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.status 2023-12-06T15:59:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-06T15:59:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=215 2023-12-06T15:59:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 215 = 215 2023-12-06T15:59:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2023-12-06T15:59:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:59:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:59:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:59:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:59:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2023-12-06T15:59:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_49': 2023-12-06T15:59:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:59:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:59:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:59:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b295d734c9 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b295d734c9 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:51:44 2023 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:51:44 2023 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:51:44 2023 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: a5b2b9b1-e05d-4ea2-ba09-82cd6039f259 2023-12-06T15:59:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:59:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_49 snap 2023-12-06T15:59:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T15:59:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:59:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:59:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_49.export 2023-12-06T15:59:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.export 2023-12-06T15:59:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_49.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.export 2023-12-06T15:59:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_49.export 2023-12-06T15:59:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:59:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.export 2023-12-06T15:59:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:59:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_49.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.export 2023-12-06T15:59:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_49.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_49.export 2023-12-06T15:59:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_50 2023-12-06T15:59:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snap cluster2 mirror image_50 snap 2023-12-06T15:59:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:59:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2023-12-06T15:59:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2023-12-06T15:59:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2023-12-06T15:59:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2023-12-06T15:59:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:59:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:59:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:59:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2023-12-06T15:59:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2023-12-06T15:59:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2023-12-06T15:59:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2023-12-06T15:59:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2023-12-06T15:59:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2023-12-06T15:59:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:59:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2023-12-06T15:59:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T15:59:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2023-12-06T15:59:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2023-12-06T15:59:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-06T15:59:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2023-12-06T15:59:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-06T15:59:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 3 2023-12-06T15:59:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:59:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2023-12-06T15:59:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2023-12-06T15:59:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:59:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:59:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-06T15:59:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2023-12-06T15:59:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:59:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:59:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:59:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T15:59:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-06T15:59:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:59:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:59:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2023-12-06T15:59:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:59:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-06T15:59:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2023-12-06T15:59:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-06T15:59:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:59:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:59:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-06T15:59:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2023-12-06T15:59:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-12-06T15:59:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2023-12-06T15:59:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2023-12-06T15:59:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-06T15:59:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2023-12-06T15:59:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2023-12-06T15:59:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2023-12-06T15:59:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2023-12-06T15:59:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-06T15:59:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2023-12-06T15:59:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2023-12-06T15:59:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2023-12-06T15:59:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T15:59:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2023-12-06T15:59:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-06T15:59:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2023-12-06T15:59:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:59:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2023-12-06T15:59:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:59:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-06T15:59:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2023-12-06T15:59:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2023-12-06T15:59:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2023-12-06T15:59:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-06T15:59:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2023-12-06T15:59:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2023-12-06T15:59:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2023-12-06T15:59:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2023-12-06T15:59:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2023-12-06T15:59:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-06T15:59:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2023-12-06T15:59:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2023-12-06T15:59:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2023-12-06T15:59:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2023-12-06T15:59:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-06T15:59:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:59:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:59:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-06T15:59:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2023-12-06T15:59:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2023-12-06T15:59:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2023-12-06T15:59:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2023-12-06T15:59:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2023-12-06T15:59:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2023-12-06T15:59:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2023-12-06T15:59:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2023-12-06T15:59:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2023-12-06T15:59:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:59:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = journal ']' 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' snapshot = snapshot ']' 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-image_50.status 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-image_50.status 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:59:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_50.status 2023-12-06T15:59:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-image_50.status 2023-12-06T15:59:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-image_50.status 2023-12-06T15:59:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-06T15:59:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_status_log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.status 2023-12-06T15:59:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2023-12-06T15:59:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2023-12-06T15:59:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Snapshot ID: 217 2023-12-06T15:59:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_50.status 2023-12-06T15:59:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_50.status 2023-12-06T15:59:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2023-12-06T15:59:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-06T15:59:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snapshot_id=217 2023-12-06T15:59:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:59:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2023-12-06T15:59:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.status 2023-12-06T15:59:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.status 2023-12-06T15:59:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-06T15:59:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=117 2023-12-06T15:59:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 217 = 117 2023-12-06T15:59:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2023-12-06T15:59:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.status 2023-12-06T15:59:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.status 2023-12-06T15:59:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-06T15:59:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:59:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:59:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:59:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:59:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:59:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 217 = '' 2023-12-06T15:59:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2023-12-06T15:59:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.status 2023-12-06T15:59:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.status 2023-12-06T15:59:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-06T15:59:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T15:59:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:-:1.1: Document is empty 2023-12-06T15:59:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2023-12-06T15:59:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:^ 2023-12-06T15:59:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id= 2023-12-06T15:59:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 217 = '' 2023-12-06T15:59:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-06T15:59:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1.6 2023-12-06T15:59:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.status 2023-12-06T15:59:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local log=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.status 2023-12-06T15:59:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-06T15:59:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-06T15:59:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-06T15:59:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local primary_snapshot_id=217 2023-12-06T15:59:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 217 = 217 2023-12-06T15:59:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2023-12-06T15:59:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T15:59:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:59:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T15:59:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-06T15:59:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2023-12-06T15:59:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2023-12-06T15:59:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_50': 2023-12-06T15:59:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2023-12-06T15:59:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T15:59:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2023-12-06T15:59:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10b24b6c6a90 2023-12-06T15:59:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10b24b6c6a90 2023-12-06T15:59:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T15:59:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-06T15:59:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T15:59:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T15:59:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 15:51:58 2023 2023-12-06T15:59:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 15:51:58 2023 2023-12-06T15:59:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 15:51:58 2023 2023-12-06T15:59:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2023-12-06T15:59:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T15:59:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T15:59:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c9d745c5-f598-4e49-aa04-df5e0f8e27d8 2023-12-06T15:59:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2023-12-06T15:59:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_snaps mirror image_50 snap 2023-12-06T15:59:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T15:59:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap 2023-12-06T15:59:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2023-12-06T15:59:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_50.export 2023-12-06T15:59:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.export 2023-12-06T15:59:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_50.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.export 2023-12-06T15:59:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_50.export 2023-12-06T15:59:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:59:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.export 2023-12-06T15:59:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-06T15:59:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_50.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.export 2023-12-06T15:59:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.Q1BJHHgE3s/cluster2-mirror-image_50.export /tmp/tmp.Q1BJHHgE3s/cluster1-mirror-image_50.export 2023-12-06T15:59:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: delete many images' 2023-12-06T15:59:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.Q1BJHHgE3s/rbd-mirror.test.log 2023-12-06T15:59:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2023-12-06T15:59:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2023-12-06 15:59:30 TEST: delete many images 2023-12-06T15:59:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2023-12-06 15:59:30 TEST: delete many images 2023-12-06T15:59:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 50 2023-12-06T15:59:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_1 2023-12-06T15:59:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2023-12-06T15:59:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:59:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T15:59:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2023-12-06T15:59:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:59:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-06T15:59:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:59:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T15:59:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T15:59:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_1 2023-12-06T15:59:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T15:59:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-06T15:59:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-06T15:59:31.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:31.396+0000 7f6abb501700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c4aa1f7380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T15:59:31.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:31.396+0000 7f6ac250f700 -1 rbd::mirror::ImageReplayer: 0x55c4a8179400 [3/0b9a1c1c-7b41-4db7-9c1f-9f9fa3190aa6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T15:59:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T15:59:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_2 2023-12-06T15:59:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2023-12-06T15:59:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:59:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T15:59:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2023-12-06T15:59:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:59:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-06T15:59:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:59:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T15:59:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T15:59:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_2 2023-12-06T15:59:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T15:59:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-06T15:59:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-06T15:59:34.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:34.408+0000 7f6abb501700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c4aa298b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T15:59:34.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:34.408+0000 7f6ac1d0e700 -1 rbd::mirror::ImageReplayer: 0x55c4a9f2c780 [3/ddc26196-98fe-465d-a33a-3c9e85520f7d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T15:59:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T15:59:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_3 2023-12-06T15:59:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2023-12-06T15:59:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:59:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T15:59:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2023-12-06T15:59:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:59:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-06T15:59:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:59:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T15:59:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T15:59:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_3 2023-12-06T15:59:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T15:59:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-06T15:59:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-06T15:59:37.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:37.421+0000 7f6abb501700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c4aa1f7520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T15:59:37.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:37.421+0000 7f6ac250f700 -1 rbd::mirror::ImageReplayer: 0x55c4a9e6e500 [3/82a11cbd-ed20-4f4a-8948-aac80e8e0a67] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T15:59:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T15:59:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_4 2023-12-06T15:59:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2023-12-06T15:59:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:59:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T15:59:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2023-12-06T15:59:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:59:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-06T15:59:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:59:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T15:59:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T15:59:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_4 2023-12-06T15:59:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T15:59:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-06T15:59:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-06T15:59:40.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:40.432+0000 7f6abb501700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c4aa2bcd00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T15:59:40.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:40.432+0000 7f6ac1d0e700 -1 rbd::mirror::ImageReplayer: 0x55c4a9e6f400 [3/b7f20ec4-f4b1-4e5b-a31f-db50e7727e1c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T15:59:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T15:59:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_5 2023-12-06T15:59:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2023-12-06T15:59:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:59:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T15:59:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2023-12-06T15:59:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:59:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-06T15:59:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:59:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T15:59:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T15:59:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_5 2023-12-06T15:59:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T15:59:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-06T15:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-06T15:59:43.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:43.444+0000 7f6abb501700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c4aa2bcea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T15:59:43.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:43.444+0000 7f6ac1d0e700 -1 rbd::mirror::ImageReplayer: 0x55c4a9e26000 [3/292f59e4-56d9-46a7-8aff-49cef92c6c9c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T15:59:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T15:59:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_6 2023-12-06T15:59:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2023-12-06T15:59:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:59:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T15:59:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2023-12-06T15:59:46.277 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5345724951287829) > weight (0.25) 2023-12-06T15:59:46.277 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-06T15:59:46.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-06T15:59:46.278 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-06T15:59:46.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi204.stderr:2023-12-06T15:59:46.277+0000 7f3a53e94700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 120850) UID: 1000 2023-12-06T15:59:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:59:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-06T15:59:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:59:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T15:59:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T15:59:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_6 2023-12-06T15:59:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T15:59:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-06T15:59:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-06T15:59:46.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T15:59:46.465+0000 7fc7c5db0700 -1 rbd::mirror::ImageMap: 0x55e70e0967e0 handle_peer_ack_remove: failed to remove global_image_id=1b242e7d-b3bc-4380-95c1-3f0e519a2478 2023-12-06T15:59:46.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T15:59:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T15:59:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_7 2023-12-06T15:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2023-12-06T15:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T15:59:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2023-12-06T15:59:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:59:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-06T15:59:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:59:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T15:59:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T15:59:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_7 2023-12-06T15:59:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T15:59:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-06T15:59:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-06T15:59:50.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:50.607+0000 7f6abbd02700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c4aa1f7860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T15:59:50.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:50.607+0000 7f6ac250f700 -1 rbd::mirror::ImageReplayer: 0x55c4a9e6ef00 [3/a45295cc-1ac0-4902-a0d3-c086b4cca650] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T15:59:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T15:59:51.279 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-06T15:59:51.279 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T15:59:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T15:59:51.279 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-06T15:59:51.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-06T15:59:51.280 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-06T15:59:51.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-06T15:59:51.282 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-12-06T15:59:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_8 2023-12-06T15:59:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2023-12-06T15:59:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:59:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T15:59:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2023-12-06T15:59:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:59:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-06T15:59:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:59:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T15:59:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T15:59:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_8 2023-12-06T15:59:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T15:59:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-06T15:59:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-06T15:59:52.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:52.483+0000 7f6abb501700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c4aa2bd040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T15:59:52.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:52.483+0000 7f6ac1d0e700 -1 rbd::mirror::ImageReplayer: 0x55c4a9f2ca00 [3/f4484446-8808-413e-bb2b-96fb61446c98] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T15:59:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T15:59:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_9 2023-12-06T15:59:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2023-12-06T15:59:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:59:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T15:59:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2023-12-06T15:59:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:59:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_9 2023-12-06T15:59:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:59:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T15:59:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T15:59:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_9 2023-12-06T15:59:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T15:59:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-06T15:59:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-06T15:59:55.458 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.453+0000 7f44ae2f8700 -1 reset not still connected to 0x559a151b4dd0 2023-12-06T15:59:55.458 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.453+0000 7f44ae2f8700 -1 reset not still connected to 0x559a151b51e0 2023-12-06T15:59:55.458 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.453+0000 7f44ae2f8700 -1 reset not still connected to 0x559a181a4dd0 2023-12-06T15:59:55.458 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.454+0000 7f44ae2f8700 -1 reset not still connected to 0x559a1ab9af70 2023-12-06T15:59:55.458 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.454+0000 7f44ae2f8700 -1 reset not still connected to 0x559a1b18a820 2023-12-06T15:59:55.458 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.454+0000 7f44ae2f8700 -1 reset not still connected to 0x559a1b18b790 2023-12-06T15:59:55.459 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.454+0000 7f44ae2f8700 -1 reset not still connected to 0x559a1d69f1e0 2023-12-06T15:59:55.459 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.454+0000 7f44ae2f8700 -1 reset not still connected to 0x559a2171c410 2023-12-06T15:59:55.459 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.454+0000 7f44ae2f8700 -1 reset not still connected to 0x559a2171c9c0 2023-12-06T15:59:55.459 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.454+0000 7f44ae2f8700 -1 reset not still connected to 0x559a2171d380 2023-12-06T15:59:55.459 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.454+0000 7f44ae2f8700 -1 reset not still connected to 0x559a2171dba0 2023-12-06T15:59:55.459 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.454+0000 7f44ae2f8700 -1 reset not still connected to 0x559a27fe4dd0 2023-12-06T15:59:55.459 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T15:59:55.454+0000 7f44ae2f8700 -1 reset not still connected to 0x559a344dd6c0 2023-12-06T15:59:55.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:55.508+0000 7f6abb501700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c4aa2bd1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T15:59:55.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:55.508+0000 7f6ac1d0e700 -1 rbd::mirror::ImageReplayer: 0x55c4a9f2d400 [3/f202fb70-81a7-49b9-b61d-6d5b735f390f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T15:59:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T15:59:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T15:59:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T15:59:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_10 2023-12-06T15:59:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2023-12-06T15:59:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:59:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T15:59:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2023-12-06T15:59:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T15:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_10 2023-12-06T15:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T15:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T15:59:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_10 2023-12-06T15:59:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T15:59:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T15:59:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T15:59:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-06T15:59:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-06T15:59:58.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:58.521+0000 7f6abbd02700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c4aa1f7a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T15:59:58.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T15:59:58.521+0000 7f6ac250f700 -1 rbd::mirror::ImageReplayer: 0x55c4a9e6e000 [3/70428fb0-ef2f-4a77-a9f3-f9c2d486736f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_11 2023-12-06T16:00:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2023-12-06T16:00:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T16:00:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2023-12-06T16:00:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_11 2023-12-06T16:00:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T16:00:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_11 2023-12-06T16:00:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T16:00:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2023-12-06T16:00:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2023-12-06T16:00:01.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:00:01.326+0000 7f6abb501700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c4aa1f7ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:01.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:00:01.326+0000 7f6ac250f700 -1 rbd::mirror::ImageReplayer: 0x55c4a9e6ea00 [3/85642e31-6be7-4259-b3a5-09a8ae17198f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_12 2023-12-06T16:00:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2023-12-06T16:00:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T16:00:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2023-12-06T16:00:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_12 2023-12-06T16:00:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T16:00:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_12 2023-12-06T16:00:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T16:00:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2023-12-06T16:00:04.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2023-12-06T16:00:04.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:00:04.334+0000 7f6abb501700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c4aa2bd380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:04.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:00:04.334+0000 7f6ac1d0e700 -1 rbd::mirror::ImageReplayer: 0x55c4a9e26500 [3/2bd42a12-9068-4ea7-b5df-749817b7f6cc] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:04.451 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-06T16:00:04.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-06T16:00:04.451 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-06T16:00:04.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:00:04.450+0000 7f6ac2d10700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 134701) UID: 1000 2023-12-06T16:00:04.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T16:00:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_13 2023-12-06T16:00:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2023-12-06T16:00:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T16:00:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2023-12-06T16:00:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_13 2023-12-06T16:00:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T16:00:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_13 2023-12-06T16:00:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T16:00:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2023-12-06T16:00:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2023-12-06T16:00:08.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:08.541+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91ec0ad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:08.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:08.560+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91ec0a680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:08.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:08.560+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb5180 [3/d04aac8e-3190-4279-b4cb-4b1a37a6b42f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_14 2023-12-06T16:00:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2023-12-06T16:00:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T16:00:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2023-12-06T16:00:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_14 2023-12-06T16:00:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T16:00:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_14 2023-12-06T16:00:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T16:00:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2023-12-06T16:00:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2023-12-06T16:00:10.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:10.359+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91cd66000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:10.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:10.359+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb6e000 [3/216c840f-8878-4153-b225-e1d08de4c3fd] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_15 2023-12-06T16:00:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2023-12-06T16:00:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T16:00:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2023-12-06T16:00:12.452 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-06T16:00:12.452 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T16:00:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T16:00:12.453 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-06T16:00:12.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-06T16:00:12.453 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-06T16:00:12.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-06T16:00:12.456 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2023-12-06T16:00:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_15 2023-12-06T16:00:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T16:00:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_15 2023-12-06T16:00:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T16:00:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2023-12-06T16:00:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2023-12-06T16:00:13.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:13.557+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91ec0a9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:13.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:13.557+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb6f680 [3/500b919a-f976-4b7b-8c44-5adea4cfc7db] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_16 2023-12-06T16:00:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2023-12-06T16:00:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T16:00:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2023-12-06T16:00:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_16 2023-12-06T16:00:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T16:00:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_16 2023-12-06T16:00:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T16:00:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2023-12-06T16:00:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2023-12-06T16:00:16.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:16.382+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91ec0ad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:16.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:16.382+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb5900 [3/dc7f2a11-16c4-4259-8d8f-2015f551ea93] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_17 2023-12-06T16:00:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2023-12-06T16:00:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T16:00:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2023-12-06T16:00:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_17 2023-12-06T16:00:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T16:00:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_17 2023-12-06T16:00:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T16:00:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2023-12-06T16:00:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2023-12-06T16:00:19.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:19.387+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91ec0a680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:19.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:19.387+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb4000 [3/b11d8c4b-991a-40fd-a328-c851d1d3afd6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_18 2023-12-06T16:00:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2023-12-06T16:00:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T16:00:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2023-12-06T16:00:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_18 2023-12-06T16:00:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T16:00:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_18 2023-12-06T16:00:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T16:00:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2023-12-06T16:00:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2023-12-06T16:00:22.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:22.397+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91ec0aea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:22.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:22.397+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb4280 [3/bf968f6f-af59-4018-a2bf-c1b130c0104d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_19 2023-12-06T16:00:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2023-12-06T16:00:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T16:00:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2023-12-06T16:00:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_19 2023-12-06T16:00:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T16:00:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_19 2023-12-06T16:00:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T16:00:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2023-12-06T16:00:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2023-12-06T16:00:25.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:25.417+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91ec0a1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:25.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:25.417+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb4a00 [3/cadad124-8d2e-48fe-a7b8-df7c14a3f2c0] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_20 2023-12-06T16:00:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2023-12-06T16:00:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T16:00:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2023-12-06T16:00:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_20 2023-12-06T16:00:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T16:00:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_20 2023-12-06T16:00:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T16:00:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2023-12-06T16:00:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2023-12-06T16:00:28.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:28.425+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e8ec9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:28.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:28.425+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb98500 [3/7c1ca29b-6451-4600-b10e-63d11596db15] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:28.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:28.591+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91ec0a1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:28.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:28.591+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb98500 [3/7c1ca29b-6451-4600-b10e-63d11596db15] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_21 2023-12-06T16:00:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2023-12-06T16:00:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T16:00:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2023-12-06T16:00:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_21 2023-12-06T16:00:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T16:00:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_21 2023-12-06T16:00:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T16:00:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2023-12-06T16:00:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2023-12-06T16:00:31.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:31.350+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91ea1fd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:31.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:31.350+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb6e280 [3/298c48a0-cc50-45ca-995e-2df71ee446f2] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_22 2023-12-06T16:00:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2023-12-06T16:00:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T16:00:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2023-12-06T16:00:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_22 2023-12-06T16:00:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T16:00:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_22 2023-12-06T16:00:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T16:00:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2023-12-06T16:00:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2023-12-06T16:00:34.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:34.450+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91ea1fa00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:34.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:34.450+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb5b80 [3/f9078af2-0a9f-4ece-8c12-87bbc54a7d15] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_23 2023-12-06T16:00:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2023-12-06T16:00:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T16:00:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2023-12-06T16:00:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_23 2023-12-06T16:00:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T16:00:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_23 2023-12-06T16:00:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T16:00:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2023-12-06T16:00:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2023-12-06T16:00:37.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:37.474+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e8ecd00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:37.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:37.474+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb4500 [3/c279a46f-dce9-48d0-be3a-726f061d7181] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_24 2023-12-06T16:00:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2023-12-06T16:00:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T16:00:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2023-12-06T16:00:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_24 2023-12-06T16:00:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T16:00:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_24 2023-12-06T16:00:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T16:00:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2023-12-06T16:00:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2023-12-06T16:00:40.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:40.472+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e8ed380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:40.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:40.472+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb98f00 [3/a3bf4bbf-7635-41c7-8700-c2fe78a3fbe0] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_25 2023-12-06T16:00:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2023-12-06T16:00:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T16:00:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2023-12-06T16:00:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_25 2023-12-06T16:00:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T16:00:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_25 2023-12-06T16:00:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T16:00:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2023-12-06T16:00:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2023-12-06T16:00:43.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:43.486+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e8ec340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:43.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:43.486+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb5400 [3/db2fb06f-c4e5-4af2-8764-3691b6d24393] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_26 2023-12-06T16:00:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2023-12-06T16:00:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T16:00:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2023-12-06T16:00:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_26 2023-12-06T16:00:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T16:00:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_26 2023-12-06T16:00:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T16:00:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2023-12-06T16:00:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2023-12-06T16:00:46.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:46.507+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e6c1a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:46.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:46.507+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb98780 [3/8f93da9b-50b8-4808-bf76-a4022502be07] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_27 2023-12-06T16:00:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2023-12-06T16:00:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T16:00:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2023-12-06T16:00:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_27 2023-12-06T16:00:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T16:00:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_27 2023-12-06T16:00:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T16:00:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2023-12-06T16:00:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2023-12-06T16:00:49.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:49.715+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e8ed1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:49.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:49.715+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb99900 [3/c0035e14-7c4a-401d-a289-cee1fa7ce091] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_28 2023-12-06T16:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2023-12-06T16:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T16:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2023-12-06T16:00:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_28 2023-12-06T16:00:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T16:00:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_28 2023-12-06T16:00:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T16:00:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2023-12-06T16:00:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2023-12-06T16:00:52.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:52.514+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e8ed040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:52.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:52.514+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb6ec80 [3/45a35e04-1f3a-4f6c-a378-30b333fab016] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_29 2023-12-06T16:00:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2023-12-06T16:00:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T16:00:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2023-12-06T16:00:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_29 2023-12-06T16:00:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T16:00:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_29 2023-12-06T16:00:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T16:00:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2023-12-06T16:00:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2023-12-06T16:00:55.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:55.508+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91eb9b860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:55.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:55.508+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91a913900 [3/0031e62d-baea-4f3c-837b-04d500983bed] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:56.817 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T16:00:56.813+0000 7f0764987700 -1 reset not still connected to 0x56287bcf9040 2023-12-06T16:00:56.817 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T16:00:56.813+0000 7f0764987700 -1 reset not still connected to 0x56287d6ff930 2023-12-06T16:00:56.817 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T16:00:56.813+0000 7f0764987700 -1 reset not still connected to 0x56287d7c75f0 2023-12-06T16:00:56.817 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T16:00:56.815+0000 7f0764987700 -1 reset not still connected to 0x56288036e410 2023-12-06T16:00:56.818 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T16:00:56.816+0000 7f0764987700 -1 reset not still connected to 0x5628814a4820 2023-12-06T16:00:56.818 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T16:00:56.816+0000 7f0764987700 -1 reset not still connected to 0x5628815cd040 2023-12-06T16:00:56.818 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T16:00:56.816+0000 7f0764987700 -1 reset not still connected to 0x562882080d00 2023-12-06T16:00:56.818 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T16:00:56.816+0000 7f0764987700 -1 reset not still connected to 0x5628820816c0 2023-12-06T16:00:56.818 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T16:00:56.816+0000 7f0764987700 -1 reset not still connected to 0x562882ad9ee0 2023-12-06T16:00:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:00:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:00:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:00:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_30 2023-12-06T16:00:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2023-12-06T16:00:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T16:00:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:00:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2023-12-06T16:00:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_30 2023-12-06T16:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T16:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:00:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:00:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_30 2023-12-06T16:00:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:00:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:00:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T16:00:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2023-12-06T16:00:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2023-12-06T16:00:58.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:58.529+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e9ead00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:58.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:58.529+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb6ea00 [3/393bd03f-74d9-4863-9753-476b06c160ad] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:00:58.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:58.591+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e9ea4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:00:58.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:00:58.591+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb6ea00 [3/393bd03f-74d9-4863-9753-476b06c160ad] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_31 2023-12-06T16:01:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2023-12-06T16:01:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T16:01:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2023-12-06T16:01:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_31 2023-12-06T16:01:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T16:01:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_31 2023-12-06T16:01:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T16:01:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2023-12-06T16:01:01.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2023-12-06T16:01:01.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:01.534+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e9ebba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:01.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:01.534+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb4780 [3/c5f2021b-abb5-4a19-80cd-cedca15b14e1] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:02.797 INFO:tasks.ceph.cluster1.osd.1.smithi176.stderr:2023-12-06T16:01:02.794+0000 7f5a0595c700 -1 reset not still connected to 0x55d5ff80c000 2023-12-06T16:01:02.797 INFO:tasks.ceph.cluster1.osd.1.smithi176.stderr:2023-12-06T16:01:02.794+0000 7f5a0595c700 -1 reset not still connected to 0x55d600ebc820 2023-12-06T16:01:02.797 INFO:tasks.ceph.cluster1.osd.1.smithi176.stderr:2023-12-06T16:01:02.794+0000 7f5a0595c700 -1 reset not still connected to 0x55d600ebd110 2023-12-06T16:01:02.797 INFO:tasks.ceph.cluster1.osd.1.smithi176.stderr:2023-12-06T16:01:02.794+0000 7f5a0595c700 -1 reset not still connected to 0x55d600ebd860 2023-12-06T16:01:02.797 INFO:tasks.ceph.cluster1.osd.1.smithi176.stderr:2023-12-06T16:01:02.794+0000 7f5a0595c700 -1 reset not still connected to 0x55d6024989c0 2023-12-06T16:01:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_32 2023-12-06T16:01:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2023-12-06T16:01:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:03.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:03.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T16:01:03.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:03.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2023-12-06T16:01:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_32 2023-12-06T16:01:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T16:01:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_32 2023-12-06T16:01:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T16:01:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2023-12-06T16:01:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2023-12-06T16:01:04.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:04.562+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91eb9a000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:04.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:04.562+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb98a00 [3/994706d1-52be-4379-8f1d-c660ef60435c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_33 2023-12-06T16:01:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2023-12-06T16:01:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T16:01:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2023-12-06T16:01:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_33 2023-12-06T16:01:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T16:01:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_33 2023-12-06T16:01:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T16:01:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2023-12-06T16:01:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2023-12-06T16:01:07.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:07.566+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91eb9a9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:07.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:07.566+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb6e780 [3/3293bbff-63fc-47b0-b7af-aa270d848ce2] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_34 2023-12-06T16:01:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2023-12-06T16:01:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T16:01:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2023-12-06T16:01:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_34 2023-12-06T16:01:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T16:01:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_34 2023-12-06T16:01:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T16:01:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2023-12-06T16:01:10.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2023-12-06T16:01:10.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:10.580+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e9ea820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:10.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:10.580+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb5680 [3/dc0720e9-3db1-4118-8a6c-3ea4239b21d7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_35 2023-12-06T16:01:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2023-12-06T16:01:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T16:01:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2023-12-06T16:01:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_35 2023-12-06T16:01:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T16:01:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_35 2023-12-06T16:01:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T16:01:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2023-12-06T16:01:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2023-12-06T16:01:13.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:13.341+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e9ea9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:13.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:13.341+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91a913b80 [3/12508ac3-20ec-4ba0-98c2-40a26628f314] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_36 2023-12-06T16:01:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2023-12-06T16:01:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T16:01:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2023-12-06T16:01:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_36 2023-12-06T16:01:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T16:01:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_36 2023-12-06T16:01:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T16:01:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2023-12-06T16:01:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2023-12-06T16:01:16.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:16.357+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91eb9a4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:16.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:16.357+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb6f900 [3/4ef4b927-2e3c-4c53-87cf-79905d0ea25b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_37 2023-12-06T16:01:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2023-12-06T16:01:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T16:01:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2023-12-06T16:01:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_37 2023-12-06T16:01:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T16:01:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_37 2023-12-06T16:01:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T16:01:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2023-12-06T16:01:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2023-12-06T16:01:19.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:19.368+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91eb9ab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:19.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:19.368+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb6fb80 [3/9a78f30b-f892-4362-84e4-ebdc27892076] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_38 2023-12-06T16:01:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2023-12-06T16:01:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T16:01:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2023-12-06T16:01:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_38 2023-12-06T16:01:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T16:01:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_38 2023-12-06T16:01:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T16:01:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2023-12-06T16:01:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2023-12-06T16:01:22.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:22.391+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91eb9a340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:22.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:22.391+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb6f400 [3/3f1f4f6b-ba96-4370-b170-e42f93596565] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_39 2023-12-06T16:01:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2023-12-06T16:01:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T16:01:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2023-12-06T16:01:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_39 2023-12-06T16:01:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T16:01:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_39 2023-12-06T16:01:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T16:01:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2023-12-06T16:01:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2023-12-06T16:01:25.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:25.403+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e9eb380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:25.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:25.403+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb98280 [3/70cf571a-cab9-45c9-bf8a-20c249f7dcf1] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_40 2023-12-06T16:01:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2023-12-06T16:01:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T16:01:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2023-12-06T16:01:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_40 2023-12-06T16:01:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T16:01:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_40 2023-12-06T16:01:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T16:01:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2023-12-06T16:01:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2023-12-06T16:01:28.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:28.421+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e9f3ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:28.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:28.421+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb99680 [3/bcd56cda-90d7-45de-b6e1-3c9183852338] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:28.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:28.584+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e9d3040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:28.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:28.584+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb99680 [3/bcd56cda-90d7-45de-b6e1-3c9183852338] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_41 2023-12-06T16:01:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2023-12-06T16:01:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T16:01:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2023-12-06T16:01:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_41 2023-12-06T16:01:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T16:01:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_41 2023-12-06T16:01:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T16:01:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2023-12-06T16:01:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2023-12-06T16:01:31.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:31.344+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e7c5040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:31.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:31.344+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91ec10000 [3/d19eb783-02bd-4211-9062-fbc83db629fb] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_42 2023-12-06T16:01:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2023-12-06T16:01:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T16:01:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2023-12-06T16:01:33.456 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9336104958398821) > weight (0.25) 2023-12-06T16:01:33.457 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7523458580796094) > weight (0.25) 2023-12-06T16:01:33.457 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4666319034015083) > weight (0.25) 2023-12-06T16:01:33.457 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5146425821576592) > weight (0.25) 2023-12-06T16:01:33.457 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2023-12-06T16:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_42 2023-12-06T16:01:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T16:01:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_42 2023-12-06T16:01:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T16:01:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2023-12-06T16:01:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2023-12-06T16:01:34.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:34.445+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e98eea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:34.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:34.445+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb6ef00 [3/4c2635cc-57ee-4102-bf76-ae809079b50d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_43 2023-12-06T16:01:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2023-12-06T16:01:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T16:01:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2023-12-06T16:01:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:37.283 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3361145850236147) > weight (0.25) 2023-12-06T16:01:37.283 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-06T16:01:37.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-06T16:01:37.284 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-06T16:01:37.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi204.stderr:2023-12-06T16:01:37.283+0000 7fcb2af39700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 148274) UID: 1000 2023-12-06T16:01:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_43 2023-12-06T16:01:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T16:01:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_43 2023-12-06T16:01:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T16:01:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2023-12-06T16:01:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2023-12-06T16:01:37.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:37.455+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e8aa4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:37.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:37.455+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb99400 [3/b9035ac9-e202-49a4-90ed-1f7ed899a231] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:37.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T16:01:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_44 2023-12-06T16:01:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2023-12-06T16:01:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T16:01:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2023-12-06T16:01:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_44 2023-12-06T16:01:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T16:01:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_44 2023-12-06T16:01:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T16:01:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2023-12-06T16:01:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2023-12-06T16:01:40.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:40.469+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e98eb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:40.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:40.469+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb98c80 [3/9a98f3b9-5d2f-447d-9e2f-0f9938f259fe] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_45 2023-12-06T16:01:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2023-12-06T16:01:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T16:01:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2023-12-06T16:01:43.284 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-06T16:01:43.285 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T16:01:43.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T16:01:43.285 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-06T16:01:43.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-06T16:01:43.285 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-06T16:01:43.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-06T16:01:43.291 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2023-12-06T16:01:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_45 2023-12-06T16:01:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T16:01:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_45 2023-12-06T16:01:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T16:01:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2023-12-06T16:01:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2023-12-06T16:01:43.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:43.505+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e98fa00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:43.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:43.505+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb98000 [3/58132a72-7f69-4342-8240-26753ebd27ff] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_46 2023-12-06T16:01:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2023-12-06T16:01:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T16:01:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2023-12-06T16:01:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_46 2023-12-06T16:01:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T16:01:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_46 2023-12-06T16:01:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T16:01:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2023-12-06T16:01:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2023-12-06T16:01:46.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:46.468+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e98e1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:46.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:46.468+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb4c80 [3/c54283d5-014c-482e-9681-a7b875a0863c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_47 2023-12-06T16:01:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2023-12-06T16:01:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T16:01:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2023-12-06T16:01:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_47 2023-12-06T16:01:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T16:01:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_47 2023-12-06T16:01:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T16:01:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2023-12-06T16:01:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2023-12-06T16:01:49.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:49.490+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e98fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:49.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:49.490+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb99b80 [3/c4567031-4494-4727-b144-9c229376da8f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_48 2023-12-06T16:01:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2023-12-06T16:01:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T16:01:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2023-12-06T16:01:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_48 2023-12-06T16:01:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T16:01:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_48 2023-12-06T16:01:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T16:01:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2023-12-06T16:01:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2023-12-06T16:01:52.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:52.702+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e98ed00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:52.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:52.702+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb6f180 [3/3ec3a26c-39fa-4d23-b6cb-9a34c415483f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_49 2023-12-06T16:01:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2023-12-06T16:01:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T16:01:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2023-12-06T16:01:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_49 2023-12-06T16:01:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T16:01:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_49 2023-12-06T16:01:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T16:01:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2023-12-06T16:01:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2023-12-06T16:01:55.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:55.500+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e98e340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:55.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:55.500+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91eb99180 [3/a5b2b9b1-e05d-4ea2-ba09-82cd6039f259] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:01:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:01:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:01:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_50 2023-12-06T16:01:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2023-12-06T16:01:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T16:01:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2023-12-06T16:01:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2023-12-06T16:01:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2023-12-06T16:01:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_50 2023-12-06T16:01:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T16:01:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:01:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:01:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_50 2023-12-06T16:01:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:01:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:01:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T16:01:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2023-12-06T16:01:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2023-12-06T16:01:58.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:58.518+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91ea29520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:58.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:58.518+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb4f00 [3/c9d745c5-f598-4e49-aa04-df5e0f8e27d8] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:01:58.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:58.582+0000 7f2ef931e700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e98e000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:01:58.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:01:58.582+0000 7f2effb2b700 -1 rbd::mirror::ImageReplayer: 0x55f91ebb4f00 [3/c9d745c5-f598-4e49-aa04-df5e0f8e27d8] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:02:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-06T16:02:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: image deletions should propagate' 2023-12-06T16:02:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.Q1BJHHgE3s/rbd-mirror.test.log 2023-12-06T16:02:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2023-12-06T16:02:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2023-12-06 16:02:00 TEST: image deletions should propagate 2023-12-06T16:02:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2023-12-06 16:02:00 TEST: image deletions should propagate 2023-12-06T16:02:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_pool_images cluster1 mirror 0 2023-12-06T16:02:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_count=0 2023-12-06T16:02:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count 2023-12-06T16:02:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_count=0 2023-12-06T16:02:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2023-12-06T16:02:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 40 2023-12-06T16:02:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in `seq 1 40` 2023-12-06T16:02:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 -ne 1 2023-12-06T16:02:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-06T16:02:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep 'images: ' 2023-12-06T16:02:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut '-d ' -f 2 2023-12-06T16:02:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T16:02:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ count=0 2023-12-06T16:02:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2023-12-06T16:02:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2023-12-06T16:02:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state 2023-12-06T16:02:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 40 2023-12-06T16:02:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in `seq 1 40` 2023-12-06T16:02:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 -ne 1 2023-12-06T16:02:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-06T16:02:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep 'image health:' 2023-12-06T16:02:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut '-d ' -f 3 2023-12-06T16:02:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: failed to query services: (13) Permission denied 2023-12-06T16:02:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=OK 2023-12-06T16:02:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test OK = ERROR 2023-12-06T16:02:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test OK = OK 2023-12-06T16:02:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 50 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_1 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_1 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_1 2023-12-06T16:02:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-06T16:02:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-06T16:02:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-12-06T16:02:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_1 2023-12-06T16:02:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_1 2023-12-06T16:02:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-06T16:02:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-06T16:02:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_2 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_2 2023-12-06T16:02:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_2 2023-12-06T16:02:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-06T16:02:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-06T16:02:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-12-06T16:02:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:02:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_2 2023-12-06T16:02:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_2 2023-12-06T16:02:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-06T16:02:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-06T16:02:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_3 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_3 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_3 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-06T16:02:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-06T16:02:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-12-06T16:02:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:02:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_3 2023-12-06T16:02:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_3 2023-12-06T16:02:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-06T16:02:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-06T16:02:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_4 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_4 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_4 2023-12-06T16:02:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-06T16:02:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-06T16:02:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-12-06T16:02:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:02:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_4 2023-12-06T16:02:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_4 2023-12-06T16:02:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-06T16:02:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:01.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-06T16:02:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_5 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_5 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_5 2023-12-06T16:02:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-06T16:02:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:01.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-06T16:02:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:01.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_5 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_5 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-06T16:02:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-06T16:02:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_6 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_6 2023-12-06T16:02:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_6 2023-12-06T16:02:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-06T16:02:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-06T16:02:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-12-06T16:02:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:02:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_6 2023-12-06T16:02:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_6 2023-12-06T16:02:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-06T16:02:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-06T16:02:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_7 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_7 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_7 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-06T16:02:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-06T16:02:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2023-12-06T16:02:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:02:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_7 2023-12-06T16:02:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_7 2023-12-06T16:02:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-06T16:02:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-06T16:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_8 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_8 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_8 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-06T16:02:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-06T16:02:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2023-12-06T16:02:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:02:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_8 2023-12-06T16:02:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_8 2023-12-06T16:02:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-06T16:02:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-06T16:02:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_9 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_9 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_9 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-06T16:02:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-06T16:02:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2023-12-06T16:02:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:02:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_9 2023-12-06T16:02:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_9 2023-12-06T16:02:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-06T16:02:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_10 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_10 2023-12-06T16:02:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_10 2023-12-06T16:02:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-06T16:02:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-06T16:02:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2023-12-06T16:02:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T16:02:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_10 2023-12-06T16:02:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_10 2023-12-06T16:02:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-06T16:02:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-06T16:02:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_11 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_11 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_11 2023-12-06T16:02:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-12-06T16:02:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-06T16:02:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2023-12-06T16:02:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_11 2023-12-06T16:02:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_11 2023-12-06T16:02:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_11 2023-12-06T16:02:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-12-06T16:02:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-06T16:02:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_12 2023-12-06T16:02:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2023-12-06T16:02:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T16:02:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_12 2023-12-06T16:02:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_12 2023-12-06T16:02:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-12-06T16:02:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-06T16:02:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2023-12-06T16:02:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_12 2023-12-06T16:02:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_12 2023-12-06T16:02:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_12 2023-12-06T16:02:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-12-06T16:02:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-06T16:02:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_13 2023-12-06T16:02:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2023-12-06T16:02:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T16:02:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_13 2023-12-06T16:02:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_13 2023-12-06T16:02:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-12-06T16:02:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-06T16:02:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2023-12-06T16:02:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_13 2023-12-06T16:02:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_13 2023-12-06T16:02:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_13 2023-12-06T16:02:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-12-06T16:02:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-06T16:02:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_14 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_14 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_14 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-12-06T16:02:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-06T16:02:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_14 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_14 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_14 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-12-06T16:02:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-06T16:02:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_15 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_15 2023-12-06T16:02:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_15 2023-12-06T16:02:02.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-12-06T16:02:02.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-06T16:02:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_15 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_15 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_15 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-12-06T16:02:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-06T16:02:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_16 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_16 2023-12-06T16:02:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_16 2023-12-06T16:02:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-12-06T16:02:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-06T16:02:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2023-12-06T16:02:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_16 2023-12-06T16:02:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_16 2023-12-06T16:02:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_16 2023-12-06T16:02:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-12-06T16:02:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-06T16:02:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_17 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_17 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_17 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-12-06T16:02:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-06T16:02:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2023-12-06T16:02:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_17 2023-12-06T16:02:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_17 2023-12-06T16:02:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_17 2023-12-06T16:02:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-12-06T16:02:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-06T16:02:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_18 2023-12-06T16:02:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2023-12-06T16:02:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T16:02:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_18 2023-12-06T16:02:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_18 2023-12-06T16:02:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-12-06T16:02:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-06T16:02:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2023-12-06T16:02:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_18 2023-12-06T16:02:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_18 2023-12-06T16:02:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_18 2023-12-06T16:02:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-12-06T16:02:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_19 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_19 2023-12-06T16:02:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_19 2023-12-06T16:02:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-12-06T16:02:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-06T16:02:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2023-12-06T16:02:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_19 2023-12-06T16:02:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_19 2023-12-06T16:02:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_19 2023-12-06T16:02:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-12-06T16:02:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_20 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_20 2023-12-06T16:02:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_20 2023-12-06T16:02:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-12-06T16:02:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-06T16:02:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_20 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_20 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_20 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-12-06T16:02:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-06T16:02:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_21 2023-12-06T16:02:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_21 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_21 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-12-06T16:02:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-06T16:02:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2023-12-06T16:02:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_21 2023-12-06T16:02:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_21 2023-12-06T16:02:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_21 2023-12-06T16:02:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-12-06T16:02:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-06T16:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_22 2023-12-06T16:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2023-12-06T16:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T16:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_22 2023-12-06T16:02:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_22 2023-12-06T16:02:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-12-06T16:02:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-06T16:02:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_22 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_22 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_22 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-12-06T16:02:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-06T16:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_23 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_23 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_23 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-12-06T16:02:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-06T16:02:04.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:04.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:04.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2023-12-06T16:02:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_23 2023-12-06T16:02:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_23 2023-12-06T16:02:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_23 2023-12-06T16:02:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-12-06T16:02:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-06T16:02:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_24 2023-12-06T16:02:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2023-12-06T16:02:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T16:02:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_24 2023-12-06T16:02:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_24 2023-12-06T16:02:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-12-06T16:02:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-06T16:02:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2023-12-06T16:02:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_24 2023-12-06T16:02:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_24 2023-12-06T16:02:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_24 2023-12-06T16:02:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-12-06T16:02:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-06T16:02:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_25 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_25 2023-12-06T16:02:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_25 2023-12-06T16:02:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-12-06T16:02:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-06T16:02:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2023-12-06T16:02:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_25 2023-12-06T16:02:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_25 2023-12-06T16:02:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_25 2023-12-06T16:02:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-12-06T16:02:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-06T16:02:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_26 2023-12-06T16:02:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2023-12-06T16:02:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T16:02:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_26 2023-12-06T16:02:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_26 2023-12-06T16:02:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-12-06T16:02:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-06T16:02:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2023-12-06T16:02:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_26 2023-12-06T16:02:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_26 2023-12-06T16:02:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_26 2023-12-06T16:02:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-12-06T16:02:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-06T16:02:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_27 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_27 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_27 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-12-06T16:02:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-06T16:02:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2023-12-06T16:02:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_27 2023-12-06T16:02:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_27 2023-12-06T16:02:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_27 2023-12-06T16:02:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-12-06T16:02:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_28 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_28 2023-12-06T16:02:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_28 2023-12-06T16:02:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-12-06T16:02:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-06T16:02:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2023-12-06T16:02:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_28 2023-12-06T16:02:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_28 2023-12-06T16:02:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_28 2023-12-06T16:02:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-12-06T16:02:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_29 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_29 2023-12-06T16:02:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_29 2023-12-06T16:02:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-12-06T16:02:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-06T16:02:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_29 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_29 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_29 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-12-06T16:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-06T16:02:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_30 2023-12-06T16:02:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_30 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_30 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-12-06T16:02:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-06T16:02:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_30 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_30 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_30 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-12-06T16:02:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-06T16:02:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_31 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_31 2023-12-06T16:02:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_31 2023-12-06T16:02:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-12-06T16:02:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-06T16:02:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2023-12-06T16:02:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_31 2023-12-06T16:02:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_31 2023-12-06T16:02:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_31 2023-12-06T16:02:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-12-06T16:02:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-06T16:02:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_32 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_32 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_32 2023-12-06T16:02:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-12-06T16:02:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-06T16:02:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2023-12-06T16:02:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_32 2023-12-06T16:02:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_32 2023-12-06T16:02:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_32 2023-12-06T16:02:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-12-06T16:02:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-06T16:02:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_33 2023-12-06T16:02:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2023-12-06T16:02:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T16:02:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_33 2023-12-06T16:02:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_33 2023-12-06T16:02:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-12-06T16:02:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-06T16:02:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2023-12-06T16:02:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_33 2023-12-06T16:02:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_33 2023-12-06T16:02:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_33 2023-12-06T16:02:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-12-06T16:02:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-06T16:02:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_34 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_34 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_34 2023-12-06T16:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-12-06T16:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-06T16:02:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2023-12-06T16:02:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_34 2023-12-06T16:02:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_34 2023-12-06T16:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_34 2023-12-06T16:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-12-06T16:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_35 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_35 2023-12-06T16:02:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_35 2023-12-06T16:02:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-12-06T16:02:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-06T16:02:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_35 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_35 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_35 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-12-06T16:02:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-06T16:02:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_36 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_36 2023-12-06T16:02:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_36 2023-12-06T16:02:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-12-06T16:02:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-06T16:02:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2023-12-06T16:02:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_36 2023-12-06T16:02:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_36 2023-12-06T16:02:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_36 2023-12-06T16:02:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-12-06T16:02:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-06T16:02:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_37 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_37 2023-12-06T16:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_37 2023-12-06T16:02:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-12-06T16:02:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-06T16:02:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2023-12-06T16:02:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_37 2023-12-06T16:02:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_37 2023-12-06T16:02:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_37 2023-12-06T16:02:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-12-06T16:02:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-06T16:02:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_38 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_38 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_38 2023-12-06T16:02:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-12-06T16:02:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:06.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-06T16:02:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2023-12-06T16:02:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_38 2023-12-06T16:02:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_38 2023-12-06T16:02:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_38 2023-12-06T16:02:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-12-06T16:02:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-06T16:02:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_39 2023-12-06T16:02:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2023-12-06T16:02:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T16:02:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_39 2023-12-06T16:02:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_39 2023-12-06T16:02:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-12-06T16:02:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-06T16:02:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_39 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_39 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_39 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-12-06T16:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-06T16:02:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_40 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_40 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_40 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-12-06T16:02:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-06T16:02:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_40 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_40 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_40 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-12-06T16:02:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-06T16:02:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_41 2023-12-06T16:02:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2023-12-06T16:02:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T16:02:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_41 2023-12-06T16:02:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_41 2023-12-06T16:02:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-12-06T16:02:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-06T16:02:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2023-12-06T16:02:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_41 2023-12-06T16:02:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_41 2023-12-06T16:02:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_41 2023-12-06T16:02:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-12-06T16:02:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-06T16:02:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_42 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_42 2023-12-06T16:02:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_42 2023-12-06T16:02:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-12-06T16:02:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-06T16:02:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2023-12-06T16:02:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_42 2023-12-06T16:02:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_42 2023-12-06T16:02:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_42 2023-12-06T16:02:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-12-06T16:02:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-06T16:02:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_43 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_43 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_43 2023-12-06T16:02:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-12-06T16:02:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-06T16:02:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2023-12-06T16:02:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_43 2023-12-06T16:02:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_43 2023-12-06T16:02:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_43 2023-12-06T16:02:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-12-06T16:02:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-06T16:02:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_44 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_44 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_44 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-12-06T16:02:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-06T16:02:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2023-12-06T16:02:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_44 2023-12-06T16:02:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_44 2023-12-06T16:02:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_44 2023-12-06T16:02:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-12-06T16:02:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_45 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_45 2023-12-06T16:02:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_45 2023-12-06T16:02:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-12-06T16:02:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-06T16:02:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2023-12-06T16:02:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_45 2023-12-06T16:02:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_45 2023-12-06T16:02:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_45 2023-12-06T16:02:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-12-06T16:02:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-06T16:02:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_46 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_46 2023-12-06T16:02:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_46 2023-12-06T16:02:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-12-06T16:02:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-06T16:02:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2023-12-06T16:02:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_46 2023-12-06T16:02:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_46 2023-12-06T16:02:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_46 2023-12-06T16:02:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-12-06T16:02:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-06T16:02:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_47 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_47 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_47 2023-12-06T16:02:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-12-06T16:02:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-06T16:02:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2023-12-06T16:02:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_47 2023-12-06T16:02:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_47 2023-12-06T16:02:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_47 2023-12-06T16:02:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-12-06T16:02:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-06T16:02:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_48 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_48 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_48 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-12-06T16:02:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-06T16:02:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2023-12-06T16:02:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_48 2023-12-06T16:02:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_48 2023-12-06T16:02:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_48 2023-12-06T16:02:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-12-06T16:02:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-06T16:02:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_49 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_49 2023-12-06T16:02:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_49 2023-12-06T16:02:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-12-06T16:02:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-06T16:02:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2023-12-06T16:02:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_49 2023-12-06T16:02:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_49 2023-12-06T16:02:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_49 2023-12-06T16:02:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-12-06T16:02:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-06T16:02:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_50 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_50 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_50 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-12-06T16:02:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-06T16:02:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2023-12-06T16:02:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_50 2023-12-06T16:02:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_50 2023-12-06T16:02:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_50 2023-12-06T16:02:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-12-06T16:02:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-06T16:02:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: delete images during bootstrap' 2023-12-06T16:02:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.Q1BJHHgE3s/rbd-mirror.test.log 2023-12-06T16:02:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2023-12-06T16:02:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2023-12-06 16:02:08 TEST: delete images during bootstrap 2023-12-06T16:02:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2023-12-06 16:02:08 TEST: delete images during bootstrap 2023-12-06T16:02:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_pool_mirror_mode cluster1 mirror image 2023-12-06T16:02:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=image 2023-12-06T16:02:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2023-12-06T16:02:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd: mirroring is already configured for image mode 2023-12-06T16:02:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_pool_mirror_mode cluster2 mirror image 2023-12-06T16:02:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:02:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=image 2023-12-06T16:02:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2023-12-06T16:02:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd: mirroring is already configured for image mode 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1 cluster instance 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=cluster1 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 = cluster1 ']' 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2023-12-06T16:02:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=test 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 10 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_1 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_1 512M 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:02:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T16:02:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 512M ']' 2023-12-06T16:02:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=512M 2023-12-06T16:02:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:02:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-12-06T16:02:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_1 2023-12-06T16:02:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:02:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T16:02:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-06T16:02:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T16:02:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-06T16:02:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_1': 2023-12-06T16:02:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T16:02:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T16:02:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 18f8f2d6f6c0 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.18f8f2d6f6c0 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 16:02:08 2023 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 16:02:08 2023 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 16:02:08 2023 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: a5d85f3b-dce1-4095-8da1-9245da37d471 2023-12-06T16:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T16:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-06T16:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T16:02:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=14 2023-12-06T16:02:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T16:02:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-06T16:02:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 82262016 bytes 2023-12-06T16:02:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 112609280 bytes 2023-12-06T16:02:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 118808576 bytes 2023-12-06T16:02:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 233160704 bytes 2023-12-06T16:02:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 246173696 bytes 2023-12-06T16:02:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 270110720 bytes 2023-12-06T16:02:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 283443200 bytes 2023-12-06T16:02:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 335708160 bytes 2023-12-06T16:02:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 354093056 bytes 2023-12-06T16:02:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 396332032 bytes 2023-12-06T16:02:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 421799936 bytes 2023-12-06T16:02:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 471948288 bytes 2023-12-06T16:02:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 495068160 bytes 2023-12-06T16:02:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 506587136 bytes 2023-12-06T16:02:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 545029120 bytes 2023-12-06T16:02:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 585717760 bytes 2023-12-06T16:02:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 630728704 bytes 2023-12-06T16:02:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 652395520 bytes 2023-12-06T16:02:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 672447488 bytes 2023-12-06T16:02:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 714604544 bytes 2023-12-06T16:02:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 741331968 bytes 2023-12-06T16:02:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 749116416 bytes 2023-12-06T16:02:13.715 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T16:02:13.713+0000 7f6ed7b3c700 -1 reset not still connected to 0x55e8f6154ea0 2023-12-06T16:02:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 780905472 bytes 2023-12-06T16:02:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 802223104 bytes 2023-12-06T16:02:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 858080256 bytes 2023-12-06T16:02:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 912814080 bytes 2023-12-06T16:02:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 913440768 bytes 2023-12-06T16:02:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 964002816 bytes 2023-12-06T16:02:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1044659200 bytes 2023-12-06T16:02:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1087942656 bytes 2023-12-06T16:02:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1125213184 bytes 2023-12-06T16:02:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1171598336 bytes 2023-12-06T16:02:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1267686400 bytes 2023-12-06T16:02:15.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1278235648 bytes 2023-12-06T16:02:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1308240896 bytes 2023-12-06T16:02:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1352206336 bytes 2023-12-06T16:02:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1363313664 bytes 2023-12-06T16:02:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1409555456 bytes 2023-12-06T16:02:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1454175232 bytes 2023-12-06T16:02:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1471569920 bytes 2023-12-06T16:02:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1497252864 bytes 2023-12-06T16:02:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1515164672 bytes 2023-12-06T16:02:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1534436352 bytes 2023-12-06T16:02:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1548340224 bytes 2023-12-06T16:02:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1609931776 bytes 2023-12-06T16:02:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1680697344 bytes 2023-12-06T16:02:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1726129152 bytes 2023-12-06T16:02:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1732721664 bytes 2023-12-06T16:02:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1786835968 bytes 2023-12-06T16:02:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1817507840 bytes 2023-12-06T16:02:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 1818137600 bytes 2023-12-06T16:02:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 1918390272 bytes 2023-12-06T16:02:19.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 1958691840 bytes 2023-12-06T16:02:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 1975482368 bytes 2023-12-06T16:02:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2034504704 bytes 2023-12-06T16:02:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2035105792 bytes 2023-12-06T16:02:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2063025152 bytes 2023-12-06T16:02:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2085642240 bytes 2023-12-06T16:02:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2134095872 bytes 2023-12-06T16:02:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2177445888 bytes 2023-12-06T16:02:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2195181568 bytes 2023-12-06T16:02:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2221494272 bytes 2023-12-06T16:02:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2279581696 bytes 2023-12-06T16:02:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2346788864 bytes 2023-12-06T16:02:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 2402829312 bytes 2023-12-06T16:02:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6600 IOs, 2491002880 bytes 2023-12-06T16:02:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6700 IOs, 2567909376 bytes 2023-12-06T16:02:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6800 IOs, 2596097024 bytes 2023-12-06T16:02:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6900 IOs, 2668503040 bytes 2023-12-06T16:02:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7000 IOs, 2699916288 bytes 2023-12-06T16:02:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7100 IOs, 2723757056 bytes 2023-12-06T16:02:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7200 IOs, 2742281216 bytes 2023-12-06T16:02:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7300 IOs, 2816594944 bytes 2023-12-06T16:02:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7400 IOs, 2858888192 bytes 2023-12-06T16:02:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_1 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_1 2023-12-06T16:02:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-06T16:02:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=12a0759d2644 2023-12-06T16:02:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_1 present 12a0759d2644 2023-12-06T16:02:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2023-12-06T16:02:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=12a0759d2644 2023-12-06T16:02:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_1 2023-12-06T16:02:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_1 2023-12-06T16:02:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-06T16:02:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=12a0759d2644 2023-12-06T16:02:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 12a0759d2644 2023-12-06T16:02:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 12a0759d2644 -o 12a0759d2644 = 12a0759d2644 2023-12-06T16:02:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2023-12-06T16:02:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2023-12-06T16:02:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror image_1 2023-12-06T16:02:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:02:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2023-12-06T16:02:28.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:02:28.582+0000 7f2ef8b1d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f91e7b0000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:02:28.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:02:28.582+0000 7f2eff32a700 -1 rbd::mirror::ImageReplayer: 0x55f91eb99180 [3/a5d85f3b-dce1-4095-8da1-9245da37d471] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:02:54.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T16:02:54.475+0000 7fc7b8d96700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e70f45fc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2023-12-06T16:02:54.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T16:02:54.475+0000 7fc7b8d96700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e70f45fc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=368, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2023-12-06T16:02:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2023-12-06T16:02:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-06T16:02:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:02:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:02:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_1 2023-12-06T16:02:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_1 2023-12-06T16:02:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-06T16:02:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-06T16:02:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:02:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:02:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:02:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_1 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_1 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-06T16:02:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:02:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-06T16:02:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:02:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:02:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:02:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ purge_snapshots cluster2 mirror image_1 2023-12-06T16:02:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:02:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2023-12-06T16:02:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-06T16:02:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:02:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:02:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:02:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_1 2023-12-06T16:02:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:02:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_1 2023-12-06T16:02:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-06T16:02:54.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-06T16:02:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-06T16:02:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_2 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_2 512M 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 512M ']' 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=512M 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:02:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-12-06T16:02:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_2 2023-12-06T16:02:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:02:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:02:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T16:02:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-06T16:02:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T16:02:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-06T16:02:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_2': 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 19107edc4102 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.19107edc4102 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 16:02:55 2023 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 16:02:55 2023 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 16:02:55 2023 2023-12-06T16:02:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T16:02:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T16:02:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 40b2e6f0-4b91-4290-b58a-519bd88f939e 2023-12-06T16:02:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T16:02:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-06T16:02:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:02:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:02:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:02:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T16:02:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=13 2023-12-06T16:02:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T16:02:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-06T16:02:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 55851008 bytes 2023-12-06T16:02:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 107401216 bytes 2023-12-06T16:02:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 173191168 bytes 2023-12-06T16:02:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 208556032 bytes 2023-12-06T16:02:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 243470336 bytes 2023-12-06T16:02:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 303220736 bytes 2023-12-06T16:02:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 343146496 bytes 2023-12-06T16:02:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 375705600 bytes 2023-12-06T16:02:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 409286656 bytes 2023-12-06T16:02:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 434493440 bytes 2023-12-06T16:02:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 458765312 bytes 2023-12-06T16:02:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 507448320 bytes 2023-12-06T16:02:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 552897536 bytes 2023-12-06T16:02:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 629569536 bytes 2023-12-06T16:02:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 661449728 bytes 2023-12-06T16:02:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 711735296 bytes 2023-12-06T16:02:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 762643456 bytes 2023-12-06T16:02:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 800579584 bytes 2023-12-06T16:02:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 808511488 bytes 2023-12-06T16:02:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 861799424 bytes 2023-12-06T16:02:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 929012736 bytes 2023-12-06T16:02:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 962621440 bytes 2023-12-06T16:03:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 963279872 bytes 2023-12-06T16:03:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1051247616 bytes 2023-12-06T16:03:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1064033280 bytes 2023-12-06T16:03:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1090618368 bytes 2023-12-06T16:03:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1151229952 bytes 2023-12-06T16:03:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1164838912 bytes 2023-12-06T16:03:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1232302080 bytes 2023-12-06T16:03:01.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1286144000 bytes 2023-12-06T16:03:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1316075520 bytes 2023-12-06T16:03:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1348357120 bytes 2023-12-06T16:03:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1369204736 bytes 2023-12-06T16:03:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1395892224 bytes 2023-12-06T16:03:02.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1469585408 bytes 2023-12-06T16:03:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1526568960 bytes 2023-12-06T16:03:02.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1620132864 bytes 2023-12-06T16:03:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1691994112 bytes 2023-12-06T16:03:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1707269120 bytes 2023-12-06T16:03:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1737224192 bytes 2023-12-06T16:03:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1779636224 bytes 2023-12-06T16:03:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1851524096 bytes 2023-12-06T16:03:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1878627328 bytes 2023-12-06T16:03:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1934008320 bytes 2023-12-06T16:03:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1969478656 bytes 2023-12-06T16:03:04.458 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7216499462041787) > weight (0.25) 2023-12-06T16:03:04.458 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-06T16:03:04.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-06T16:03:04.458 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-06T16:03:04.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:03:04.457+0000 7f2f0032c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 131221) UID: 1000 2023-12-06T16:03:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1998049280 bytes 2023-12-06T16:03:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2058954752 bytes 2023-12-06T16:03:04.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T16:03:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2094192640 bytes 2023-12-06T16:03:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2106784768 bytes 2023-12-06T16:03:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2126099456 bytes 2023-12-06T16:03:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2129012736 bytes 2023-12-06T16:03:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2187783168 bytes 2023-12-06T16:03:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2208970752 bytes 2023-12-06T16:03:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2244083712 bytes 2023-12-06T16:03:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2343812096 bytes 2023-12-06T16:03:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2367226880 bytes 2023-12-06T16:03:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2504729600 bytes 2023-12-06T16:03:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2569191424 bytes 2023-12-06T16:03:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2657388544 bytes 2023-12-06T16:03:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2675958784 bytes 2023-12-06T16:03:07.459 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-06T16:03:07.459 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T16:03:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T16:03:07.460 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-06T16:03:07.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-06T16:03:07.460 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-06T16:03:07.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-06T16:03:07.463 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2023-12-06T16:03:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2701630464 bytes 2023-12-06T16:03:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2736588800 bytes 2023-12-06T16:03:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2785699840 bytes 2023-12-06T16:03:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2892969984 bytes 2023-12-06T16:03:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 2920521728 bytes 2023-12-06T16:03:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T16:03:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T16:03:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T16:03:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:03:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2023-12-06T16:03:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:03:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:03:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2023-12-06T16:03:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:03:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:03:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:03:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_2 2023-12-06T16:03:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:03:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:03:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_2 2023-12-06T16:03:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-06T16:03:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:03:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=12a04f32c487 2023-12-06T16:03:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:03:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:03:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_2 present 12a04f32c487 2023-12-06T16:03:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:03:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:03:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2023-12-06T16:03:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=12a04f32c487 2023-12-06T16:03:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:03:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:03:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_2 2023-12-06T16:03:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:03:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:03:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_2 2023-12-06T16:03:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-06T16:03:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:03:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=12a04f32c487 2023-12-06T16:03:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 12a04f32c487 2023-12-06T16:03:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 12a04f32c487 -o 12a04f32c487 = 12a04f32c487 2023-12-06T16:03:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2023-12-06T16:03:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2023-12-06T16:03:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:03:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror image_2 2023-12-06T16:03:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:03:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:03:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2023-12-06T16:03:12.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:03:12.582+0000 7f58ddbdc700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55598f7c8680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:03:12.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:03:12.582+0000 7f58e3be8700 -1 rbd::mirror::ImageReplayer: 0x55598b985400 [3/40b2e6f0-4b91-4290-b58a-519bd88f939e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:03:39.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T16:03:39.611+0000 7fc7b8d96700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e70f62fc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2023-12-06T16:03:39.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T16:03:39.611+0000 7fc7b8d96700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e70f62fc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=370, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2023-12-06T16:03:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2023-12-06T16:03:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-06T16:03:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:03:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:03:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:03:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:03:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:03:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:03:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_2 2023-12-06T16:03:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:03:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:03:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_2 2023-12-06T16:03:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-06T16:03:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:03:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-06T16:03:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:03:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:03:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:03:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-12-06T16:03:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:03:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:03:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:03:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:03:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:03:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:03:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_2 2023-12-06T16:03:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:03:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:03:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_2 2023-12-06T16:03:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-06T16:03:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:03:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-06T16:03:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:03:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:03:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:03:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:03:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ purge_snapshots cluster2 mirror image_2 2023-12-06T16:03:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:03:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:03:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2023-12-06T16:03:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-06T16:03:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:03:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:03:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:03:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:03:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_2 2023-12-06T16:03:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:03:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_2 2023-12-06T16:03:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-06T16:03:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-06T16:03:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-06T16:03:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:03:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T16:03:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_3 2023-12-06T16:03:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_3 512M 2023-12-06T16:03:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:03:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:03:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:03:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:03:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:03:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T16:03:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 512M ']' 2023-12-06T16:03:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=512M 2023-12-06T16:03:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:03:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-12-06T16:03:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_3 2023-12-06T16:03:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:03:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:03:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T16:03:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-06T16:03:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T16:03:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-06T16:03:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_3': 2023-12-06T16:03:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T16:03:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T16:03:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T16:03:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 19329fa4f84b 2023-12-06T16:03:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.19329fa4f84b 2023-12-06T16:03:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T16:03:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T16:03:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T16:03:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T16:03:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 16:03:40 2023 2023-12-06T16:03:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 16:03:40 2023 2023-12-06T16:03:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 16:03:40 2023 2023-12-06T16:03:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T16:03:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T16:03:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: ee6bdffe-a61e-4a23-a046-d1631cd16cb4 2023-12-06T16:03:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T16:03:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-06T16:03:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:03:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:03:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T16:03:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=8 2023-12-06T16:03:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T16:03:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-06T16:03:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 17345536 bytes 2023-12-06T16:03:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 28974080 bytes 2023-12-06T16:03:41.292 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.29140500854514084) > weight (0.25) 2023-12-06T16:03:41.292 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7939717882601647) > weight (0.25) 2023-12-06T16:03:41.292 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-06T16:03:41.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-06T16:03:41.293 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-06T16:03:41.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T16:03:41.292+0000 7fc7c65b1700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 131333) UID: 1000 2023-12-06T16:03:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 75393024 bytes 2023-12-06T16:03:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 124613632 bytes 2023-12-06T16:03:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 127537152 bytes 2023-12-06T16:03:41.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T16:03:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 192152576 bytes 2023-12-06T16:03:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 211790848 bytes 2023-12-06T16:03:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 258118656 bytes 2023-12-06T16:03:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 309846016 bytes 2023-12-06T16:03:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 310450176 bytes 2023-12-06T16:03:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 384965632 bytes 2023-12-06T16:03:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 409268224 bytes 2023-12-06T16:03:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 450253824 bytes 2023-12-06T16:03:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 495097856 bytes 2023-12-06T16:03:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 498837504 bytes 2023-12-06T16:03:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 503785472 bytes 2023-12-06T16:03:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 520325120 bytes 2023-12-06T16:03:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 570525696 bytes 2023-12-06T16:03:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 618550272 bytes 2023-12-06T16:03:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 641339392 bytes 2023-12-06T16:03:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 673957888 bytes 2023-12-06T16:03:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 690263040 bytes 2023-12-06T16:03:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 706660352 bytes 2023-12-06T16:03:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 752739328 bytes 2023-12-06T16:03:45.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 783096832 bytes 2023-12-06T16:03:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 798916608 bytes 2023-12-06T16:03:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 818087936 bytes 2023-12-06T16:03:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 871416832 bytes 2023-12-06T16:03:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 995353600 bytes 2023-12-06T16:03:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 995976192 bytes 2023-12-06T16:03:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1033308160 bytes 2023-12-06T16:03:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1064711168 bytes 2023-12-06T16:03:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1110738944 bytes 2023-12-06T16:03:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1120942080 bytes 2023-12-06T16:03:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1137162240 bytes 2023-12-06T16:03:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1157571584 bytes 2023-12-06T16:03:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1201285120 bytes 2023-12-06T16:03:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1201909760 bytes 2023-12-06T16:03:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1251743744 bytes 2023-12-06T16:03:47.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1278568448 bytes 2023-12-06T16:03:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1324048384 bytes 2023-12-06T16:03:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1367376896 bytes 2023-12-06T16:03:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1435498496 bytes 2023-12-06T16:03:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1476665344 bytes 2023-12-06T16:03:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1514615808 bytes 2023-12-06T16:03:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T16:03:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_3 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_3 2023-12-06T16:03:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-06T16:03:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:03:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=155d4409f934 2023-12-06T16:03:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:03:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:03:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_3 present 155d4409f934 2023-12-06T16:03:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:03:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:03:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2023-12-06T16:03:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=155d4409f934 2023-12-06T16:03:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:03:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:03:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_3 2023-12-06T16:03:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:03:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:03:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_3 2023-12-06T16:03:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-06T16:03:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:03:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=155d4409f934 2023-12-06T16:03:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 155d4409f934 2023-12-06T16:03:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 155d4409f934 -o 155d4409f934 = 155d4409f934 2023-12-06T16:03:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2023-12-06T16:03:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2023-12-06T16:03:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:03:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror image_3 2023-12-06T16:03:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:03:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:03:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:03:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2023-12-06T16:03:50.293 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-06T16:03:50.293 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T16:03:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T16:03:50.294 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-06T16:03:50.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-06T16:03:50.294 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-06T16:03:50.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-06T16:03:50.297 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-12-06T16:04:12.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:04:12.581+0000 7f58ddbdc700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55598f889040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:04:12.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:04:12.581+0000 7f58e3be8700 -1 rbd::mirror::ImageReplayer: 0x55598b985180 [3/ee6bdffe-a61e-4a23-a046-d1631cd16cb4] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:04:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T16:04:18.712+0000 7feaeb349700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5622baeb6000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2023-12-06T16:04:18.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T16:04:18.712+0000 7feaeb349700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5622baeb6000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2023-12-06T16:04:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_3 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_3 2023-12-06T16:04:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-06T16:04:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:04:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-06T16:04:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:04:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:04:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:04:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-12-06T16:04:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:04:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:04:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:04:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:04:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:04:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:04:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_3 2023-12-06T16:04:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:04:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:04:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_3 2023-12-06T16:04:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-06T16:04:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-06T16:04:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:04:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:04:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:04:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:04:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ purge_snapshots cluster2 mirror image_3 2023-12-06T16:04:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:04:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2023-12-06T16:04:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-06T16:04:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:04:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:04:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:04:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_3 2023-12-06T16:04:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_3 2023-12-06T16:04:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-06T16:04:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-06T16:04:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-06T16:04:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_4 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_4 512M 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 512M ']' 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=512M 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:04:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-12-06T16:04:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_4 2023-12-06T16:04:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:04:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T16:04:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-06T16:04:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T16:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-06T16:04:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_4': 2023-12-06T16:04:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T16:04:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T16:04:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T16:04:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 1950ae4eaf8d 2023-12-06T16:04:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.1950ae4eaf8d 2023-12-06T16:04:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T16:04:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T16:04:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T16:04:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T16:04:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 16:04:19 2023 2023-12-06T16:04:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 16:04:19 2023 2023-12-06T16:04:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 16:04:19 2023 2023-12-06T16:04:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T16:04:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T16:04:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 6e18e839-a2ac-4367-b785-0fb6afd446a9 2023-12-06T16:04:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T16:04:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-06T16:04:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:04:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T16:04:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=8 2023-12-06T16:04:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T16:04:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-06T16:04:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 18530304 bytes 2023-12-06T16:04:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 73154560 bytes 2023-12-06T16:04:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 135667712 bytes 2023-12-06T16:04:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 166922240 bytes 2023-12-06T16:04:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 209517568 bytes 2023-12-06T16:04:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 215269376 bytes 2023-12-06T16:04:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 286856192 bytes 2023-12-06T16:04:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 318244864 bytes 2023-12-06T16:04:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 346284032 bytes 2023-12-06T16:04:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 407180288 bytes 2023-12-06T16:04:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 425790464 bytes 2023-12-06T16:04:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 484852736 bytes 2023-12-06T16:04:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 527346688 bytes 2023-12-06T16:04:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 551524352 bytes 2023-12-06T16:04:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 595161088 bytes 2023-12-06T16:04:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 595778560 bytes 2023-12-06T16:04:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 682782720 bytes 2023-12-06T16:04:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 694296576 bytes 2023-12-06T16:04:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 713520128 bytes 2023-12-06T16:04:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 767495168 bytes 2023-12-06T16:04:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 818202624 bytes 2023-12-06T16:04:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 834017280 bytes 2023-12-06T16:04:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 910040064 bytes 2023-12-06T16:04:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 977204224 bytes 2023-12-06T16:04:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1040337920 bytes 2023-12-06T16:04:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1056782336 bytes 2023-12-06T16:04:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1120474112 bytes 2023-12-06T16:04:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1165540352 bytes 2023-12-06T16:04:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1172557824 bytes 2023-12-06T16:04:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1211881472 bytes 2023-12-06T16:04:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1271344128 bytes 2023-12-06T16:04:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1310140416 bytes 2023-12-06T16:04:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1344896000 bytes 2023-12-06T16:04:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1387045888 bytes 2023-12-06T16:04:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1452455936 bytes 2023-12-06T16:04:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1519085568 bytes 2023-12-06T16:04:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1561709568 bytes 2023-12-06T16:04:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1608135680 bytes 2023-12-06T16:04:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1622715392 bytes 2023-12-06T16:04:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1695702016 bytes 2023-12-06T16:04:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1720597504 bytes 2023-12-06T16:04:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T16:04:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T16:04:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T16:04:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:04:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2023-12-06T16:04:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:04:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:04:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2023-12-06T16:04:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:04:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:04:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:04:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_4 2023-12-06T16:04:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:04:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:04:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_4 2023-12-06T16:04:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-06T16:04:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:04:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=155dafe29965 2023-12-06T16:04:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:04:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:04:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_4 present 155dafe29965 2023-12-06T16:04:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:04:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:04:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2023-12-06T16:04:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=155dafe29965 2023-12-06T16:04:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:04:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:04:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_4 2023-12-06T16:04:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:04:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:04:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_4 2023-12-06T16:04:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-06T16:04:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:04:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=155dafe29965 2023-12-06T16:04:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 155dafe29965 2023-12-06T16:04:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 155dafe29965 -o 155dafe29965 = 155dafe29965 2023-12-06T16:04:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2023-12-06T16:04:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2023-12-06T16:04:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:04:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror image_4 2023-12-06T16:04:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:04:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2023-12-06T16:04:42.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:04:42.580+0000 7f58dd3db700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55598f8896c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:04:42.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:04:42.580+0000 7f58e3be8700 -1 rbd::mirror::ImageReplayer: 0x55598b985900 [3/6e18e839-a2ac-4367-b785-0fb6afd446a9] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:04:56.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T16:04:56.303+0000 7feaebb4a700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5622bad81800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2023-12-06T16:04:56.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T16:04:56.303+0000 7feaebb4a700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5622bad81800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2023-12-06T16:04:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2023-12-06T16:04:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-06T16:04:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:04:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:04:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:04:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:04:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:04:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:04:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_4 2023-12-06T16:04:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:04:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:04:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_4 2023-12-06T16:04:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-06T16:04:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:04:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-06T16:04:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:04:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:04:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:04:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-12-06T16:04:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:04:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_4 2023-12-06T16:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_4 2023-12-06T16:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-06T16:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:04:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-06T16:04:56.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:04:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:04:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:04:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:04:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ purge_snapshots cluster2 mirror image_4 2023-12-06T16:04:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:04:56.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2023-12-06T16:04:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-06T16:04:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:04:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:04:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:04:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_4 2023-12-06T16:04:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_4 2023-12-06T16:04:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-06T16:04:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-06T16:04:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-06T16:04:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:04:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T16:04:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_5 2023-12-06T16:04:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_5 512M 2023-12-06T16:04:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:04:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:04:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:04:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:04:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T16:04:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 512M ']' 2023-12-06T16:04:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=512M 2023-12-06T16:04:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:04:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-12-06T16:04:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_5 2023-12-06T16:04:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:04:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T16:04:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-06T16:04:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T16:04:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-06T16:04:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_5': 2023-12-06T16:04:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T16:04:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T16:04:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T16:04:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 1968cc47f917 2023-12-06T16:04:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.1968cc47f917 2023-12-06T16:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T16:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T16:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T16:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T16:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 16:04:57 2023 2023-12-06T16:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 16:04:57 2023 2023-12-06T16:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 16:04:57 2023 2023-12-06T16:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T16:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T16:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 8439cc88-b4e8-4c05-bb2a-8665c8f3895b 2023-12-06T16:04:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T16:04:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-06T16:04:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:04:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:04:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:04:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T16:04:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=9 2023-12-06T16:04:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T16:04:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-06T16:04:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 37158912 bytes 2023-12-06T16:04:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 95450112 bytes 2023-12-06T16:04:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 125868032 bytes 2023-12-06T16:04:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 180331520 bytes 2023-12-06T16:04:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 211838976 bytes 2023-12-06T16:04:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 250132480 bytes 2023-12-06T16:04:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 349472768 bytes 2023-12-06T16:04:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 350759936 bytes 2023-12-06T16:04:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 419836928 bytes 2023-12-06T16:04:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 426954752 bytes 2023-12-06T16:04:59.463 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5119731377162711) > weight (0.25) 2023-12-06T16:04:59.463 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-06T16:04:59.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-06T16:04:59.464 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-06T16:04:59.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:04:59.463+0000 7fa8abc3a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 155212) UID: 1000 2023-12-06T16:04:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 515216384 bytes 2023-12-06T16:04:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 564517888 bytes 2023-12-06T16:04:59.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T16:04:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 592216064 bytes 2023-12-06T16:05:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 649827328 bytes 2023-12-06T16:05:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 669810688 bytes 2023-12-06T16:05:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 751973376 bytes 2023-12-06T16:05:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 876298240 bytes 2023-12-06T16:05:01.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 938002432 bytes 2023-12-06T16:05:01.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 978980864 bytes 2023-12-06T16:05:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 1028595712 bytes 2023-12-06T16:05:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 1074340864 bytes 2023-12-06T16:05:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1111631872 bytes 2023-12-06T16:05:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1149875200 bytes 2023-12-06T16:05:02.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1199273984 bytes 2023-12-06T16:05:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1201682432 bytes 2023-12-06T16:05:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1262228480 bytes 2023-12-06T16:05:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1331377152 bytes 2023-12-06T16:05:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1351089152 bytes 2023-12-06T16:05:03.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1379959808 bytes 2023-12-06T16:05:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1407689728 bytes 2023-12-06T16:05:03.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1432476672 bytes 2023-12-06T16:05:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1458240512 bytes 2023-12-06T16:05:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1498492928 bytes 2023-12-06T16:05:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1557499904 bytes 2023-12-06T16:05:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1647474688 bytes 2023-12-06T16:05:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1649911808 bytes 2023-12-06T16:05:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1678736384 bytes 2023-12-06T16:05:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1720345600 bytes 2023-12-06T16:05:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1834109952 bytes 2023-12-06T16:05:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1888739328 bytes 2023-12-06T16:05:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1924804608 bytes 2023-12-06T16:05:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1935046656 bytes 2023-12-06T16:05:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1956059136 bytes 2023-12-06T16:05:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1968390144 bytes 2023-12-06T16:05:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1991182336 bytes 2023-12-06T16:05:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T16:05:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T16:05:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T16:05:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:05:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2023-12-06T16:05:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:05:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2023-12-06T16:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_5 2023-12-06T16:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_5 2023-12-06T16:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-06T16:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:05:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=155da05cc8a0 2023-12-06T16:05:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:05:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:05:06.465 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-06T16:05:06.465 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T16:05:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T16:05:06.466 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-06T16:05:06.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-06T16:05:06.466 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-06T16:05:06.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-06T16:05:06.469 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2023-12-06T16:05:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_5 present 155da05cc8a0 2023-12-06T16:05:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:05:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:05:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2023-12-06T16:05:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=155da05cc8a0 2023-12-06T16:05:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:05:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:05:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_5 2023-12-06T16:05:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:05:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:05:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_5 2023-12-06T16:05:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-06T16:05:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:05:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=155da05cc8a0 2023-12-06T16:05:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 155da05cc8a0 2023-12-06T16:05:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 155da05cc8a0 -o 155da05cc8a0 = 155da05cc8a0 2023-12-06T16:05:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2023-12-06T16:05:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2023-12-06T16:05:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:05:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror image_5 2023-12-06T16:05:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:05:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:05:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2023-12-06T16:05:12.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:05:12.582+0000 7f58ddbdc700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55598f9f7380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:05:12.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:05:12.582+0000 7f58e3be8700 -1 rbd::mirror::ImageReplayer: 0x55598b985180 [3/8439cc88-b4e8-4c05-bb2a-8665c8f3895b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:05:34.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T16:05:34.372+0000 7feaeb349700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5622bade4800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2023-12-06T16:05:34.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T16:05:34.372+0000 7feaeb349700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5622bade4800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2023-12-06T16:05:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2023-12-06T16:05:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-06T16:05:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:05:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:05:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:05:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:05:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:05:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:05:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_5 2023-12-06T16:05:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:05:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:05:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_5 2023-12-06T16:05:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-06T16:05:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:05:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-06T16:05:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:05:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:05:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:05:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-12-06T16:05:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:05:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:05:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:05:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:05:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:05:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:05:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_5 2023-12-06T16:05:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:05:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:05:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_5 2023-12-06T16:05:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-06T16:05:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:05:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-06T16:05:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:05:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:05:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:05:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:05:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ purge_snapshots cluster2 mirror image_5 2023-12-06T16:05:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:05:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:05:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2023-12-06T16:05:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-06T16:05:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:05:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:05:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:05:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:05:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_5 2023-12-06T16:05:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:05:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_5 2023-12-06T16:05:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-06T16:05:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-06T16:05:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-06T16:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T16:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_6 2023-12-06T16:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_6 512M 2023-12-06T16:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T16:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 512M ']' 2023-12-06T16:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=512M 2023-12-06T16:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-12-06T16:05:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_6 2023-12-06T16:05:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:05:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:05:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T16:05:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-06T16:05:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T16:05:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-06T16:05:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_6': 2023-12-06T16:05:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T16:05:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T16:05:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T16:05:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 198a642e2181 2023-12-06T16:05:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.198a642e2181 2023-12-06T16:05:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T16:05:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T16:05:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T16:05:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T16:05:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 16:05:34 2023 2023-12-06T16:05:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 16:05:34 2023 2023-12-06T16:05:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 16:05:34 2023 2023-12-06T16:05:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T16:05:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T16:05:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 4fbf45ee-f027-419d-b8aa-925f9d1f56b8 2023-12-06T16:05:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T16:05:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-06T16:05:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:05:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:05:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T16:05:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=13 2023-12-06T16:05:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T16:05:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-06T16:05:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 44845056 bytes 2023-12-06T16:05:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 97291264 bytes 2023-12-06T16:05:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 139067392 bytes 2023-12-06T16:05:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 178808832 bytes 2023-12-06T16:05:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 236067840 bytes 2023-12-06T16:05:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 350897152 bytes 2023-12-06T16:05:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 391682048 bytes 2023-12-06T16:05:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 452063232 bytes 2023-12-06T16:05:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 502437888 bytes 2023-12-06T16:05:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 542618624 bytes 2023-12-06T16:05:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 604049408 bytes 2023-12-06T16:05:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 646445056 bytes 2023-12-06T16:05:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 667641856 bytes 2023-12-06T16:05:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 686187520 bytes 2023-12-06T16:05:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 691862528 bytes 2023-12-06T16:05:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 811129856 bytes 2023-12-06T16:05:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 898838528 bytes 2023-12-06T16:05:39.308 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T16:05:39.305+0000 7f6ee1406700 -1 bdev(0x55e8d8dd4400 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-06T16:05:39.311 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f6ee1406700 time 2023-12-06T16:05:39.307474+0000 2023-12-06T16:05:39.311 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55e8cc1b8fce] 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x55e8ccd26874] 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55e8ccd31d11] 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f6eed2811ca] 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 5: clone() 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:*** Caught signal (Aborted) ** 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: in thread 7f6ee1406700 thread_name:bstore_aio 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f6eed28bcf0] 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2: gsignal() 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 3: abort() 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55e8cc1b909f] 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x55e8ccd26874] 2023-12-06T16:05:39.312 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55e8ccd31d11] 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f6eed2811ca] 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 8: clone() 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T16:05:39.307+0000 7f6ee1406700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f6ee1406700 time 2023-12-06T16:05:39.307474+0000 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55e8cc1b8fce] 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x55e8ccd26874] 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55e8ccd31d11] 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f6eed2811ca] 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 5: clone() 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2023-12-06T16:05:39.309+0000 7f6ee1406700 -1 *** Caught signal (Aborted) ** 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: in thread 7f6ee1406700 thread_name:bstore_aio 2023-12-06T16:05:39.313 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2023-12-06T16:05:39.314 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:05:39.314 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f6eed28bcf0] 2023-12-06T16:05:39.314 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2: gsignal() 2023-12-06T16:05:39.314 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 3: abort() 2023-12-06T16:05:39.314 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55e8cc1b909f] 2023-12-06T16:05:39.314 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x55e8ccd26874] 2023-12-06T16:05:39.314 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55e8ccd31d11] 2023-12-06T16:05:39.314 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f6eed2811ca] 2023-12-06T16:05:39.314 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 8: clone() 2023-12-06T16:05:39.314 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-06T16:05:39.314 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2023-12-06T16:05:39.327 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: -704> 2023-12-06T16:05:39.305+0000 7f6ee1406700 -1 bdev(0x55e8d8dd4400 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-06T16:05:39.327 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: -112> 2023-12-06T16:05:39.307+0000 7f6ee1406700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f6ee1406700 time 2023-12-06T16:05:39.307474+0000 2023-12-06T16:05:39.327 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-06T16:05:39.327 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2023-12-06T16:05:39.327 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55e8cc1b8fce] 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x55e8ccd26874] 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55e8ccd31d11] 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f6eed2811ca] 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 5: clone() 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 0> 2023-12-06T16:05:39.309+0000 7f6ee1406700 -1 *** Caught signal (Aborted) ** 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: in thread 7f6ee1406700 thread_name:bstore_aio 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f6eed28bcf0] 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2: gsignal() 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 3: abort() 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55e8cc1b909f] 2023-12-06T16:05:39.328 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x55e8ccd26874] 2023-12-06T16:05:39.329 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55e8ccd31d11] 2023-12-06T16:05:39.329 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f6eed2811ca] 2023-12-06T16:05:39.329 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 8: clone() 2023-12-06T16:05:39.329 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-06T16:05:39.329 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2023-12-06T16:05:39.337 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: -986> 2023-12-06T16:05:39.305+0000 7f6ee1406700 -1 bdev(0x55e8d8dd4400 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: -394> 2023-12-06T16:05:39.307+0000 7f6ee1406700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f6ee1406700 time 2023-12-06T16:05:39.307474+0000 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55e8cc1b8fce] 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x55e8ccd26874] 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55e8ccd31d11] 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f6eed2811ca] 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 5: clone() 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: -282> 2023-12-06T16:05:39.309+0000 7f6ee1406700 -1 *** Caught signal (Aborted) ** 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: in thread 7f6ee1406700 thread_name:bstore_aio 2023-12-06T16:05:39.338 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2023-12-06T16:05:39.339 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:05:39.339 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f6eed28bcf0] 2023-12-06T16:05:39.339 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2: gsignal() 2023-12-06T16:05:39.339 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 3: abort() 2023-12-06T16:05:39.339 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55e8cc1b909f] 2023-12-06T16:05:39.339 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x55e8ccd26874] 2023-12-06T16:05:39.339 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55e8ccd31d11] 2023-12-06T16:05:39.339 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f6eed2811ca] 2023-12-06T16:05:39.339 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 8: clone() 2023-12-06T16:05:39.339 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-06T16:05:39.339 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr: 2023-12-06T16:05:39.498 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:daemon-helper: command failed with exit status 5 2023-12-06T16:05:40.298 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-06T16:05:40.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-06T16:05:40.299 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-06T16:05:40.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:2023-12-06T16:05:40.297+0000 7feaf9365700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 143877) UID: 1000 2023-12-06T16:05:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 937396224 bytes 2023-12-06T16:05:41.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T16:05:41.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 993540096 bytes 2023-12-06T16:05:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 1037918208 bytes 2023-12-06T16:05:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 1109315584 bytes 2023-12-06T16:05:41.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-12-06T16:05:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 1145810944 bytes 2023-12-06T16:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 1211041792 bytes 2023-12-06T16:05:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 1238343680 bytes 2023-12-06T16:05:42.300 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-06T16:05:42.300 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T16:05:42.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T16:05:42.300 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-06T16:05:42.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-06T16:05:42.300 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-06T16:05:42.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-06T16:05:42.303 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2023-12-06T16:05:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1270866944 bytes 2023-12-06T16:05:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1316503552 bytes 2023-12-06T16:05:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1348867072 bytes 2023-12-06T16:05:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1369708544 bytes 2023-12-06T16:05:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1433359360 bytes 2023-12-06T16:05:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1470785536 bytes 2023-12-06T16:05:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1579895808 bytes 2023-12-06T16:05:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1594321920 bytes 2023-12-06T16:05:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1634523136 bytes 2023-12-06T16:05:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1699236864 bytes 2023-12-06T16:05:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1722345472 bytes 2023-12-06T16:05:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1790649344 bytes 2023-12-06T16:05:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1886469120 bytes 2023-12-06T16:05:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1964378112 bytes 2023-12-06T16:05:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 2057547776 bytes 2023-12-06T16:05:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 2069042176 bytes 2023-12-06T16:05:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 2094305280 bytes 2023-12-06T16:05:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 2150605824 bytes 2023-12-06T16:05:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 2207850496 bytes 2023-12-06T16:05:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 2252969984 bytes 2023-12-06T16:05:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 2296453120 bytes 2023-12-06T16:05:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 2314125312 bytes 2023-12-06T16:05:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2337246208 bytes 2023-12-06T16:05:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2352493568 bytes 2023-12-06T16:05:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2387479552 bytes 2023-12-06T16:05:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2463170560 bytes 2023-12-06T16:05:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2494928896 bytes 2023-12-06T16:05:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2609632256 bytes 2023-12-06T16:05:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2680040448 bytes 2023-12-06T16:05:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2703698944 bytes 2023-12-06T16:05:47.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2733878272 bytes 2023-12-06T16:05:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2771695616 bytes 2023-12-06T16:05:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2807187456 bytes 2023-12-06T16:05:47.348 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-12-06T16:05:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2882653184 bytes 2023-12-06T16:05:47.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2922743808 bytes 2023-12-06T16:05:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2984939520 bytes 2023-12-06T16:05:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 3007097856 bytes 2023-12-06T16:05:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 3048698880 bytes 2023-12-06T16:05:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 3054625792 bytes 2023-12-06T16:05:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 3055264768 bytes 2023-12-06T16:05:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 3067460608 bytes 2023-12-06T16:05:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T16:05:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T16:05:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T16:05:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:05:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2023-12-06T16:05:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:05:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:05:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2023-12-06T16:05:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:05:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:05:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:05:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_6 2023-12-06T16:05:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:05:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:05:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_6 2023-12-06T16:05:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-06T16:05:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:05:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=155db1049282 2023-12-06T16:05:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:05:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:05:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_6 present 155db1049282 2023-12-06T16:05:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:05:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:05:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2023-12-06T16:05:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=155db1049282 2023-12-06T16:05:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:05:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:05:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_6 2023-12-06T16:05:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:05:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:05:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_6 2023-12-06T16:05:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-06T16:05:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:05:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=155db1049282 2023-12-06T16:05:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 155db1049282 2023-12-06T16:05:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 155db1049282 -o 155db1049282 = 155db1049282 2023-12-06T16:05:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2023-12-06T16:05:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2023-12-06T16:05:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:05:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror image_6 2023-12-06T16:05:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:05:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:05:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:05:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2023-12-06T16:05:52.752 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-12-06T16:05:58.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-12-06T16:06:03.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-12-06T16:06:08.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-12-06T16:06:12.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:06:12.582+0000 7f58ddbdc700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55598f889520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:06:12.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:06:12.582+0000 7f58e43e9700 -1 rbd::mirror::ImageReplayer: 0x55598b985400 [3/4fbf45ee-f027-419d-b8aa-925f9d1f56b8] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:06:14.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-12-06T16:06:19.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:06:19.514+0000 7fbf3261c700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c186c400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2023-12-06T16:06:19.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:06:19.514+0000 7fbf3261c700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c186c400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2023-12-06T16:06:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2023-12-06T16:06:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-06T16:06:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:06:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:06:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:06:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:06:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:06:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:06:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_6 2023-12-06T16:06:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:06:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:06:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_6 2023-12-06T16:06:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-06T16:06:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:06:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-06T16:06:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:06:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:06:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:06:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-12-06T16:06:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:06:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:06:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:06:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:06:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:06:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:06:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_6 2023-12-06T16:06:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:06:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:06:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_6 2023-12-06T16:06:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-06T16:06:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:06:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-06T16:06:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:06:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:06:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:06:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:06:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ purge_snapshots cluster2 mirror image_6 2023-12-06T16:06:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:06:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:06:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2023-12-06T16:06:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-06T16:06:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:06:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:06:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:06:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:06:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_6 2023-12-06T16:06:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:06:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_6 2023-12-06T16:06:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-06T16:06:19.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-12-06T16:06:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-06T16:06:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-06T16:06:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:06:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T16:06:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_7 2023-12-06T16:06:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_7 512M 2023-12-06T16:06:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:06:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:06:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:06:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:06:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:06:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T16:06:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 512M ']' 2023-12-06T16:06:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=512M 2023-12-06T16:06:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:06:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-12-06T16:06:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_7 2023-12-06T16:06:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:06:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:06:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T16:06:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-06T16:06:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T16:06:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-06T16:06:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_7': 2023-12-06T16:06:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T16:06:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T16:06:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T16:06:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 19a8c54e3824 2023-12-06T16:06:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.19a8c54e3824 2023-12-06T16:06:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T16:06:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T16:06:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T16:06:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T16:06:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 16:06:20 2023 2023-12-06T16:06:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 16:06:20 2023 2023-12-06T16:06:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 16:06:20 2023 2023-12-06T16:06:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T16:06:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T16:06:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 1764e769-21e9-4d7e-896d-945964e8816e 2023-12-06T16:06:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T16:06:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-06T16:06:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:06:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:06:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T16:06:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=8 2023-12-06T16:06:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T16:06:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-06T16:06:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 36306944 bytes 2023-12-06T16:06:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 51876864 bytes 2023-12-06T16:06:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 125534208 bytes 2023-12-06T16:06:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 171771904 bytes 2023-12-06T16:06:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 209293312 bytes 2023-12-06T16:06:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 235284480 bytes 2023-12-06T16:06:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 269480960 bytes 2023-12-06T16:06:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 300800000 bytes 2023-12-06T16:06:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 359103488 bytes 2023-12-06T16:06:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 432261120 bytes 2023-12-06T16:06:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 493574144 bytes 2023-12-06T16:06:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 537814016 bytes 2023-12-06T16:06:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 586253312 bytes 2023-12-06T16:06:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 624543744 bytes 2023-12-06T16:06:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 638063616 bytes 2023-12-06T16:06:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 651388928 bytes 2023-12-06T16:06:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 674091008 bytes 2023-12-06T16:06:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 704714752 bytes 2023-12-06T16:06:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 718583808 bytes 2023-12-06T16:06:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 754323456 bytes 2023-12-06T16:06:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 813252608 bytes 2023-12-06T16:06:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 874428416 bytes 2023-12-06T16:06:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 898362368 bytes 2023-12-06T16:06:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 918892544 bytes 2023-12-06T16:06:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 933294080 bytes 2023-12-06T16:06:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1015353344 bytes 2023-12-06T16:06:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1091402752 bytes 2023-12-06T16:06:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1111450624 bytes 2023-12-06T16:06:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1126089728 bytes 2023-12-06T16:06:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1170515968 bytes 2023-12-06T16:06:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1207783424 bytes 2023-12-06T16:06:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1280609280 bytes 2023-12-06T16:06:25.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-12-06T16:06:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1310299136 bytes 2023-12-06T16:06:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1354905600 bytes 2023-12-06T16:06:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1412647936 bytes 2023-12-06T16:06:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1461087232 bytes 2023-12-06T16:06:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1551110144 bytes 2023-12-06T16:06:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1566638080 bytes 2023-12-06T16:06:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1576032256 bytes 2023-12-06T16:06:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1609754624 bytes 2023-12-06T16:06:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1667930112 bytes 2023-12-06T16:06:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1710938112 bytes 2023-12-06T16:06:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1758420992 bytes 2023-12-06T16:06:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1786693632 bytes 2023-12-06T16:06:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1812854784 bytes 2023-12-06T16:06:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1830354944 bytes 2023-12-06T16:06:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1902009344 bytes 2023-12-06T16:06:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1920510976 bytes 2023-12-06T16:06:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1973443584 bytes 2023-12-06T16:06:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2029418496 bytes 2023-12-06T16:06:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2042365952 bytes 2023-12-06T16:06:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2112501760 bytes 2023-12-06T16:06:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2121454592 bytes 2023-12-06T16:06:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2141333504 bytes 2023-12-06T16:06:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2151664640 bytes 2023-12-06T16:06:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2152284160 bytes 2023-12-06T16:06:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2208049152 bytes 2023-12-06T16:06:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2258780160 bytes 2023-12-06T16:06:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T16:06:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T16:06:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T16:06:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_7 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_7 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-06T16:06:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:06:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10caa2347576 2023-12-06T16:06:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:06:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:06:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_7 present 10caa2347576 2023-12-06T16:06:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:06:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:06:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2023-12-06T16:06:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10caa2347576 2023-12-06T16:06:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:06:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:06:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_7 2023-12-06T16:06:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:06:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:06:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_7 2023-12-06T16:06:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-06T16:06:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:06:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10caa2347576 2023-12-06T16:06:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10caa2347576 2023-12-06T16:06:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10caa2347576 -o 10caa2347576 = 10caa2347576 2023-12-06T16:06:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2023-12-06T16:06:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2023-12-06T16:06:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:06:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror image_7 2023-12-06T16:06:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:06:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:06:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2023-12-06T16:06:30.469 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9335677422845029) > weight (0.25) 2023-12-06T16:06:30.470 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-06T16:06:30.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-06T16:06:30.470 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-06T16:06:30.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:06:30.470+0000 7f5f982f5700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 156265) UID: 1000 2023-12-06T16:06:30.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-12-06T16:06:30.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T16:06:33.470 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-06T16:06:33.471 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T16:06:33.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T16:06:33.471 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-06T16:06:33.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-06T16:06:33.472 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-06T16:06:33.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-06T16:06:33.476 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2023-12-06T16:06:35.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-12-06T16:06:41.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-12-06T16:06:42.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:06:42.581+0000 7f58ddbdc700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55598c64bba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:06:42.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:06:42.581+0000 7f58e43e9700 -1 rbd::mirror::ImageReplayer: 0x55598b985b80 [3/1764e769-21e9-4d7e-896d-945964e8816e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:06:46.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-12-06T16:06:52.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-12-06T16:06:57.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-12-06T16:06:59.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:06:59.590+0000 7fbf3261c700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c16ed000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2023-12-06T16:06:59.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:06:59.590+0000 7fbf3261c700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c16ed000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2023-12-06T16:06:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2023-12-06T16:06:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-06T16:06:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:06:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:06:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:06:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:06:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:06:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:06:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_7 2023-12-06T16:06:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:06:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:06:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_7 2023-12-06T16:06:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-06T16:06:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:06:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-06T16:06:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:06:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:06:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:06:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2023-12-06T16:06:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:06:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:06:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:06:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:06:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:06:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:06:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_7 2023-12-06T16:06:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:06:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:06:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_7 2023-12-06T16:06:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-06T16:06:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:06:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-06T16:06:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:06:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:06:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:06:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:06:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ purge_snapshots cluster2 mirror image_7 2023-12-06T16:06:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:06:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:06:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2023-12-06T16:06:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-06T16:06:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:06:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:06:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:06:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:06:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_7 2023-12-06T16:06:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:06:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:06:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_7 2023-12-06T16:06:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-06T16:06:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-06T16:07:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-06T16:07:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:07:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T16:07:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_8 2023-12-06T16:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_8 512M 2023-12-06T16:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T16:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 512M ']' 2023-12-06T16:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=512M 2023-12-06T16:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:07:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-12-06T16:07:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_8 2023-12-06T16:07:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:07:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:07:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T16:07:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-06T16:07:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T16:07:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_8': 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 19c964e75bd9 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.19c964e75bd9 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 16:07:00 2023 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 16:07:00 2023 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 16:07:00 2023 2023-12-06T16:07:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T16:07:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T16:07:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: d22ddc1c-9135-4d11-b46a-9e1670ecfcde 2023-12-06T16:07:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T16:07:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-06T16:07:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:07:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:07:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T16:07:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=9 2023-12-06T16:07:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T16:07:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-06T16:07:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 27193344 bytes 2023-12-06T16:07:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 132895744 bytes 2023-12-06T16:07:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 193280000 bytes 2023-12-06T16:07:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 206093312 bytes 2023-12-06T16:07:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 217003008 bytes 2023-12-06T16:07:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 232828928 bytes 2023-12-06T16:07:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 294659072 bytes 2023-12-06T16:07:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 295274496 bytes 2023-12-06T16:07:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 372627456 bytes 2023-12-06T16:07:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 401613824 bytes 2023-12-06T16:07:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 429059072 bytes 2023-12-06T16:07:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 527405056 bytes 2023-12-06T16:07:02.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 635210752 bytes 2023-12-06T16:07:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 644065280 bytes 2023-12-06T16:07:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 671537152 bytes 2023-12-06T16:07:03.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-12-06T16:07:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 690895872 bytes 2023-12-06T16:07:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 737122304 bytes 2023-12-06T16:07:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 798408704 bytes 2023-12-06T16:07:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 819441664 bytes 2023-12-06T16:07:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 820043776 bytes 2023-12-06T16:07:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 861190144 bytes 2023-12-06T16:07:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 874106880 bytes 2023-12-06T16:07:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 895672320 bytes 2023-12-06T16:07:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 976986112 bytes 2023-12-06T16:07:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 991684608 bytes 2023-12-06T16:07:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1003101184 bytes 2023-12-06T16:07:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1039078400 bytes 2023-12-06T16:07:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1086750720 bytes 2023-12-06T16:07:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1147839488 bytes 2023-12-06T16:07:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1158922240 bytes 2023-12-06T16:07:04.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1165382656 bytes 2023-12-06T16:07:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1185693696 bytes 2023-12-06T16:07:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1250224128 bytes 2023-12-06T16:07:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1288624128 bytes 2023-12-06T16:07:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1305462784 bytes 2023-12-06T16:07:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1306057728 bytes 2023-12-06T16:07:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1379059712 bytes 2023-12-06T16:07:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1426760704 bytes 2023-12-06T16:07:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1440825344 bytes 2023-12-06T16:07:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1442173952 bytes 2023-12-06T16:07:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1525817344 bytes 2023-12-06T16:07:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1567597568 bytes 2023-12-06T16:07:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1627983872 bytes 2023-12-06T16:07:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1693034496 bytes 2023-12-06T16:07:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1703208960 bytes 2023-12-06T16:07:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 1732556800 bytes 2023-12-06T16:07:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 1749799936 bytes 2023-12-06T16:07:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 1766220800 bytes 2023-12-06T16:07:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 1805254656 bytes 2023-12-06T16:07:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 1842000896 bytes 2023-12-06T16:07:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 1870365696 bytes 2023-12-06T16:07:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 1911899136 bytes 2023-12-06T16:07:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 1925763072 bytes 2023-12-06T16:07:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 1946790912 bytes 2023-12-06T16:07:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 1982845952 bytes 2023-12-06T16:07:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 1983454208 bytes 2023-12-06T16:07:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2001075200 bytes 2023-12-06T16:07:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2013189120 bytes 2023-12-06T16:07:08.409 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2023-12-06T16:07:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2146646016 bytes 2023-12-06T16:07:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2181675008 bytes 2023-12-06T16:07:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2227208192 bytes 2023-12-06T16:07:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2281693184 bytes 2023-12-06T16:07:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2355182592 bytes 2023-12-06T16:07:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 2431013888 bytes 2023-12-06T16:07:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 2471164928 bytes 2023-12-06T16:07:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6600 IOs, 2493271040 bytes 2023-12-06T16:07:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T16:07:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_8 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:07:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_8 2023-12-06T16:07:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-06T16:07:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:07:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10caf7215523 2023-12-06T16:07:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:07:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:07:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_8 present 10caf7215523 2023-12-06T16:07:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:07:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:07:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2023-12-06T16:07:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10caf7215523 2023-12-06T16:07:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:07:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:07:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_8 2023-12-06T16:07:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:07:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:07:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_8 2023-12-06T16:07:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-06T16:07:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:07:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10caf7215523 2023-12-06T16:07:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10caf7215523 2023-12-06T16:07:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10caf7215523 -o 10caf7215523 = 10caf7215523 2023-12-06T16:07:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2023-12-06T16:07:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2023-12-06T16:07:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:07:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror image_8 2023-12-06T16:07:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:07:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:07:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2023-12-06T16:07:12.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:07:12.582+0000 7f58ddbdc700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55598c64a680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:07:12.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:07:12.582+0000 7f58e43e9700 -1 rbd::mirror::ImageReplayer: 0x55598f9f4280 [3/d22ddc1c-9135-4d11-b46a-9e1670ecfcde] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:07:13.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-12-06T16:07:19.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-12-06T16:07:24.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-12-06T16:07:30.025 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-12-06T16:07:35.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2023-12-06T16:07:36.303 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4643132769709828) > weight (0.25) 2023-12-06T16:07:36.304 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.25572389687188757) > weight (0.25) 2023-12-06T16:07:36.304 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-06T16:07:36.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-06T16:07:36.304 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-06T16:07:36.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:2023-12-06T16:07:36.304+0000 7f2f5651d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 155668) UID: 1000 2023-12-06T16:07:36.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T16:07:39.305 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-06T16:07:39.305 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T16:07:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T16:07:39.306 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-06T16:07:39.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-06T16:07:39.306 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-06T16:07:39.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-06T16:07:39.309 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-12-06T16:07:39.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:07:39.710+0000 7fbf33e1f700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c17d4400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2023-12-06T16:07:39.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:07:39.710+0000 7fbf33e1f700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c17d4400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2023-12-06T16:07:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2023-12-06T16:07:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-06T16:07:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:07:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:07:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:07:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:07:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:07:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:07:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_8 2023-12-06T16:07:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:07:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:07:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_8 2023-12-06T16:07:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-06T16:07:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:07:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-06T16:07:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:07:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:07:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:07:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2023-12-06T16:07:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:07:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:07:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:07:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:07:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:07:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:07:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_8 2023-12-06T16:07:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:07:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:07:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_8 2023-12-06T16:07:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-06T16:07:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:07:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-06T16:07:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:07:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:07:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:07:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:07:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ purge_snapshots cluster2 mirror image_8 2023-12-06T16:07:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:07:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:07:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2023-12-06T16:07:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-06T16:07:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:07:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:07:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:07:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:07:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_8 2023-12-06T16:07:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:07:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_8 2023-12-06T16:07:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-06T16:07:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-06T16:07:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-06T16:07:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:07:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T16:07:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_9 2023-12-06T16:07:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_9 512M 2023-12-06T16:07:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:07:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:07:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:07:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:07:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:07:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T16:07:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 512M ']' 2023-12-06T16:07:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=512M 2023-12-06T16:07:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:07:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2023-12-06T16:07:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_9 2023-12-06T16:07:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:07:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:07:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T16:07:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-06T16:07:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T16:07:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-06T16:07:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_9': 2023-12-06T16:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T16:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T16:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T16:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 19e79c157bf9 2023-12-06T16:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.19e79c157bf9 2023-12-06T16:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T16:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T16:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T16:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T16:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 16:07:40 2023 2023-12-06T16:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 16:07:40 2023 2023-12-06T16:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 16:07:40 2023 2023-12-06T16:07:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T16:07:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T16:07:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: ba859d0e-20d1-47c7-9a47-99d876a31f00 2023-12-06T16:07:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T16:07:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-06T16:07:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:07:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:07:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T16:07:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=14 2023-12-06T16:07:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T16:07:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-06T16:07:40.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-12-06T16:07:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 84293632 bytes 2023-12-06T16:07:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 84915200 bytes 2023-12-06T16:07:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 136508416 bytes 2023-12-06T16:07:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 182895616 bytes 2023-12-06T16:07:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 229870592 bytes 2023-12-06T16:07:41.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 266285056 bytes 2023-12-06T16:07:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 280719360 bytes 2023-12-06T16:07:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 301480960 bytes 2023-12-06T16:07:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 329000960 bytes 2023-12-06T16:07:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 371207168 bytes 2023-12-06T16:07:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 389103616 bytes 2023-12-06T16:07:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 461362176 bytes 2023-12-06T16:07:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 568113152 bytes 2023-12-06T16:07:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 613402624 bytes 2023-12-06T16:07:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 648733696 bytes 2023-12-06T16:07:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1600 IOs, 664215552 bytes 2023-12-06T16:07:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1700 IOs, 726647808 bytes 2023-12-06T16:07:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1800 IOs, 743127040 bytes 2023-12-06T16:07:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1900 IOs, 787899392 bytes 2023-12-06T16:07:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2000 IOs, 793703424 bytes 2023-12-06T16:07:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2100 IOs, 870388736 bytes 2023-12-06T16:07:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2200 IOs, 889678848 bytes 2023-12-06T16:07:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2300 IOs, 900727808 bytes 2023-12-06T16:07:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2400 IOs, 961192960 bytes 2023-12-06T16:07:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2500 IOs, 1045133312 bytes 2023-12-06T16:07:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2600 IOs, 1074210816 bytes 2023-12-06T16:07:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2700 IOs, 1129723904 bytes 2023-12-06T16:07:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2800 IOs, 1162685440 bytes 2023-12-06T16:07:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:2900 IOs, 1218557952 bytes 2023-12-06T16:07:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3000 IOs, 1309078528 bytes 2023-12-06T16:07:44.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3100 IOs, 1316003840 bytes 2023-12-06T16:07:45.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3200 IOs, 1343585280 bytes 2023-12-06T16:07:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3300 IOs, 1375155200 bytes 2023-12-06T16:07:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3400 IOs, 1418343424 bytes 2023-12-06T16:07:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3500 IOs, 1439008768 bytes 2023-12-06T16:07:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3600 IOs, 1464162304 bytes 2023-12-06T16:07:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3700 IOs, 1492000768 bytes 2023-12-06T16:07:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3800 IOs, 1608383488 bytes 2023-12-06T16:07:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:3900 IOs, 1751515136 bytes 2023-12-06T16:07:46.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-12-06T16:07:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4000 IOs, 1783274496 bytes 2023-12-06T16:07:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4100 IOs, 1851838464 bytes 2023-12-06T16:07:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4200 IOs, 1899171840 bytes 2023-12-06T16:07:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4300 IOs, 1927014400 bytes 2023-12-06T16:07:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4400 IOs, 1947087872 bytes 2023-12-06T16:07:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4500 IOs, 1991226368 bytes 2023-12-06T16:07:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4600 IOs, 2023796736 bytes 2023-12-06T16:07:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4700 IOs, 2065647616 bytes 2023-12-06T16:07:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4800 IOs, 2105373696 bytes 2023-12-06T16:07:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:4900 IOs, 2186024960 bytes 2023-12-06T16:07:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5000 IOs, 2276004864 bytes 2023-12-06T16:07:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5100 IOs, 2333706240 bytes 2023-12-06T16:07:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5200 IOs, 2347322368 bytes 2023-12-06T16:07:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5300 IOs, 2404285440 bytes 2023-12-06T16:07:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5400 IOs, 2417134592 bytes 2023-12-06T16:07:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5500 IOs, 2423137280 bytes 2023-12-06T16:07:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5600 IOs, 2507047936 bytes 2023-12-06T16:07:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5700 IOs, 2532023296 bytes 2023-12-06T16:07:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5800 IOs, 2564212736 bytes 2023-12-06T16:07:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:5900 IOs, 2618110976 bytes 2023-12-06T16:07:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6000 IOs, 2711682048 bytes 2023-12-06T16:07:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6100 IOs, 2843010048 bytes 2023-12-06T16:07:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6200 IOs, 2892505088 bytes 2023-12-06T16:07:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6300 IOs, 2979432448 bytes 2023-12-06T16:07:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6400 IOs, 3009924096 bytes 2023-12-06T16:07:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6500 IOs, 3049296896 bytes 2023-12-06T16:07:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6600 IOs, 3139869696 bytes 2023-12-06T16:07:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6700 IOs, 3171365888 bytes 2023-12-06T16:07:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6800 IOs, 3176032256 bytes 2023-12-06T16:07:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:6900 IOs, 3176649728 bytes 2023-12-06T16:07:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7000 IOs, 3190174720 bytes 2023-12-06T16:07:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7100 IOs, 3210095616 bytes 2023-12-06T16:07:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7200 IOs, 3243465728 bytes 2023-12-06T16:07:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7300 IOs, 3268642816 bytes 2023-12-06T16:07:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7400 IOs, 3280952320 bytes 2023-12-06T16:07:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7500 IOs, 3300824064 bytes 2023-12-06T16:07:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7600 IOs, 3315743744 bytes 2023-12-06T16:07:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7700 IOs, 3316360192 bytes 2023-12-06T16:07:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7800 IOs, 3355109376 bytes 2023-12-06T16:07:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:7900 IOs, 3384690688 bytes 2023-12-06T16:07:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8000 IOs, 3422376960 bytes 2023-12-06T16:07:51.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-12-06T16:07:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8100 IOs, 3471694848 bytes 2023-12-06T16:07:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8200 IOs, 3511336960 bytes 2023-12-06T16:07:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8300 IOs, 3555185664 bytes 2023-12-06T16:07:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8400 IOs, 3557078016 bytes 2023-12-06T16:07:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8500 IOs, 3655288832 bytes 2023-12-06T16:07:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8600 IOs, 3678543872 bytes 2023-12-06T16:07:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8700 IOs, 3694686208 bytes 2023-12-06T16:07:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8800 IOs, 3695301632 bytes 2023-12-06T16:07:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:8900 IOs, 3728306176 bytes 2023-12-06T16:07:52.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:9000 IOs, 3787436032 bytes 2023-12-06T16:07:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:9100 IOs, 3808375808 bytes 2023-12-06T16:07:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:9200 IOs, 3874030592 bytes 2023-12-06T16:07:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:9300 IOs, 3923885056 bytes 2023-12-06T16:07:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:9400 IOs, 3953521664 bytes 2023-12-06T16:07:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:9500 IOs, 4038684672 bytes 2023-12-06T16:07:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:9600 IOs, 4116900864 bytes 2023-12-06T16:07:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:9700 IOs, 4162414592 bytes 2023-12-06T16:07:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:9800 IOs, 4174562304 bytes 2023-12-06T16:07:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:9900 IOs, 4219160576 bytes 2023-12-06T16:07:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:10000 IOs, 4255580160 bytes 2023-12-06T16:07:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:10100 IOs, 4291201024 bytes 2023-12-06T16:07:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:10200 IOs, 4356841472 bytes 2023-12-06T16:07:54.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:10300 IOs, 4376686592 bytes 2023-12-06T16:07:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:10400 IOs, 4412428288 bytes 2023-12-06T16:07:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T16:07:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T16:07:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T16:07:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:07:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2023-12-06T16:07:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:07:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:07:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2023-12-06T16:07:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:07:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:07:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:07:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_9 2023-12-06T16:07:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:07:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:07:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_9 2023-12-06T16:07:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-06T16:07:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:07:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10ca5d21c7b2 2023-12-06T16:07:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:07:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:07:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_9 present 10ca5d21c7b2 2023-12-06T16:07:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:07:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:07:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2023-12-06T16:07:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10ca5d21c7b2 2023-12-06T16:07:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:07:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:07:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_9 2023-12-06T16:07:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:07:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:07:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_9 2023-12-06T16:07:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-06T16:07:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:07:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10ca5d21c7b2 2023-12-06T16:07:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10ca5d21c7b2 2023-12-06T16:07:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10ca5d21c7b2 -o 10ca5d21c7b2 = 10ca5d21c7b2 2023-12-06T16:07:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2023-12-06T16:07:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2023-12-06T16:07:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:07:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror image_9 2023-12-06T16:07:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:07:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:07:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:07:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2023-12-06T16:07:57.050 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-12-06T16:08:02.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2023-12-06T16:08:07.858 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-12-06T16:08:12.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:08:12.582+0000 7f58dd3db700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55598c64bba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:08:12.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:08:12.582+0000 7f58e43e9700 -1 rbd::mirror::ImageReplayer: 0x55598f9f4500 [3/ba859d0e-20d1-47c7-9a47-99d876a31f00] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:08:13.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-12-06T16:08:18.477 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-06T16:08:18.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-06T16:08:18.478 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-06T16:08:18.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:2023-12-06T16:08:18.478+0000 7f58e4bea700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 148841) UID: 1000 2023-12-06T16:08:18.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-12-06T16:08:18.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi204.stderr:daemon-helper: command failed with exit status 1 2023-12-06T16:08:22.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:08:22.529+0000 7fa86c1c9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55bce1395520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-06T16:08:22.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi204.stderr:2023-12-06T16:08:22.529+0000 7fa8729d6700 -1 rbd::mirror::ImageReplayer: 0x55bcdfac1400 [3/ba859d0e-20d1-47c7-9a47-99d876a31f00] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-06T16:08:24.070 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-12-06T16:08:24.478 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-06T16:08:24.478 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T16:08:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T16:08:24.479 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-06T16:08:24.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-06T16:08:24.479 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-06T16:08:24.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-06T16:08:24.486 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2023-12-06T16:08:25.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:08:25.853+0000 7fbf33e1f700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c1a24c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2023-12-06T16:08:25.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:08:25.853+0000 7fbf33e1f700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c1a24c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2023-12-06T16:08:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2023-12-06T16:08:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-12-06T16:08:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:08:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:08:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:08:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2023-12-06T16:08:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:08:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:08:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:08:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_9 2023-12-06T16:08:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:08:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:08:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_9 2023-12-06T16:08:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-06T16:08:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:08:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-06T16:08:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2023-12-06T16:08:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:08:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:08:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2023-12-06T16:08:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:08:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:08:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:08:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2023-12-06T16:08:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:08:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:08:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:08:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_9 2023-12-06T16:08:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:08:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:08:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_9 2023-12-06T16:08:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-06T16:08:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:08:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-06T16:08:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2023-12-06T16:08:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:08:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2023-12-06T16:08:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:08:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ purge_snapshots cluster2 mirror image_9 2023-12-06T16:08:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:08:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:08:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:08:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2023-12-06T16:08:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror image_9 2023-12-06T16:08:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:08:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:08:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:08:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-06T16:08:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2023-12-06T16:08:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror image_9 2023-12-06T16:08:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:08:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:08:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_9 2023-12-06T16:08:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-06T16:08:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-06T16:08:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-06T16:08:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 10` 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=image_10 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror image_10 512M 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 512M ']' 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=512M 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2023-12-06T16:08:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2023-12-06T16:08:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror image_10 2023-12-06T16:08:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:08:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:08:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T16:08:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=snapshot 2023-12-06T16:08:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-06T16:08:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2023-12-06T16:08:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-06T16:08:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'image_10': 2023-12-06T16:08:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 512 MiB in 128 objects 2023-12-06T16:08:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2023-12-06T16:08:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2023-12-06T16:08:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 1a07b56d9577 2023-12-06T16:08:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.1a07b56d9577 2023-12-06T16:08:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2023-12-06T16:08:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-06T16:08:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2023-12-06T16:08:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2023-12-06T16:08:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Wed Dec 6 16:08:26 2023 2023-12-06T16:08:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Wed Dec 6 16:08:26 2023 2023-12-06T16:08:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Wed Dec 6 16:08:26 2023 2023-12-06T16:08:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2023-12-06T16:08:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: snapshot 2023-12-06T16:08:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 9399994d-08ea-4773-bc50-399a00c0767d 2023-12-06T16:08:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2023-12-06T16:08:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-06T16:08:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:08:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:08:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T16:08:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-06T16:08:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local duration=7 2023-12-06T16:08:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2023-12-06T16:08:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-06T16:08:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:100 IOs, 45369344 bytes 2023-12-06T16:08:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:200 IOs, 75885568 bytes 2023-12-06T16:08:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:300 IOs, 134661120 bytes 2023-12-06T16:08:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:400 IOs, 200808448 bytes 2023-12-06T16:08:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:500 IOs, 232816640 bytes 2023-12-06T16:08:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:600 IOs, 253009920 bytes 2023-12-06T16:08:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:700 IOs, 277508096 bytes 2023-12-06T16:08:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:800 IOs, 357037056 bytes 2023-12-06T16:08:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:900 IOs, 373791744 bytes 2023-12-06T16:08:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1000 IOs, 419287040 bytes 2023-12-06T16:08:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1100 IOs, 456457216 bytes 2023-12-06T16:08:29.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1200 IOs, 475650048 bytes 2023-12-06T16:08:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1300 IOs, 541987840 bytes 2023-12-06T16:08:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1400 IOs, 612714496 bytes 2023-12-06T16:08:29.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-12-06T16:08:29.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:1500 IOs, 684225536 bytes 2023-12-06T16:08:29.640 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2023-12-06T16:08:29.638+0000 7f646256f700 -1 bdev(0x5594eb8d2400 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-06T16:08:29.642 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f646256f700 time 2023-12-06T16:08:29.639248+0000 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x5594e07ecfce] 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x5594e135a874] 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5594e1365d11] 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f646e3ea1ca] 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 5: clone() 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2023-12-06T16:08:29.640+0000 7f646256f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f646256f700 time 2023-12-06T16:08:29.639248+0000 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x5594e07ecfce] 2023-12-06T16:08:29.643 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x5594e135a874] 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5594e1365d11] 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f646e3ea1ca] 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 5: clone() 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:*** Caught signal (Aborted) ** 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: in thread 7f646256f700 thread_name:bstore_aio 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f646e3f4cf0] 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2: gsignal() 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 3: abort() 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x5594e07ed09f] 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x5594e135a874] 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5594e1365d11] 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f646e3ea1ca] 2023-12-06T16:08:29.644 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 8: clone() 2023-12-06T16:08:29.645 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2023-12-06T16:08:29.642+0000 7f646256f700 -1 *** Caught signal (Aborted) ** 2023-12-06T16:08:29.645 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: in thread 7f646256f700 thread_name:bstore_aio 2023-12-06T16:08:29.645 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2023-12-06T16:08:29.645 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:08:29.645 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f646e3f4cf0] 2023-12-06T16:08:29.645 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2: gsignal() 2023-12-06T16:08:29.645 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 3: abort() 2023-12-06T16:08:29.645 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x5594e07ed09f] 2023-12-06T16:08:29.645 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x5594e135a874] 2023-12-06T16:08:29.645 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5594e1365d11] 2023-12-06T16:08:29.645 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f646e3ea1ca] 2023-12-06T16:08:29.645 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 8: clone() 2023-12-06T16:08:29.646 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-06T16:08:29.646 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2023-12-06T16:08:29.662 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: -1528> 2023-12-06T16:08:29.638+0000 7f646256f700 -1 bdev(0x5594eb8d2400 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-06T16:08:29.663 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: -802> 2023-12-06T16:08:29.640+0000 7f646256f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f646256f700 time 2023-12-06T16:08:29.639248+0000 2023-12-06T16:08:29.663 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-06T16:08:29.664 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2023-12-06T16:08:29.664 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:08:29.664 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x5594e07ecfce] 2023-12-06T16:08:29.664 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x5594e135a874] 2023-12-06T16:08:29.664 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5594e1365d11] 2023-12-06T16:08:29.664 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f646e3ea1ca] 2023-12-06T16:08:29.664 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 5: clone() 2023-12-06T16:08:29.664 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2023-12-06T16:08:29.664 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 0> 2023-12-06T16:08:29.642+0000 7f646256f700 -1 *** Caught signal (Aborted) ** 2023-12-06T16:08:29.664 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: in thread 7f646256f700 thread_name:bstore_aio 2023-12-06T16:08:29.665 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2023-12-06T16:08:29.665 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:08:29.665 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f646e3f4cf0] 2023-12-06T16:08:29.665 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2: gsignal() 2023-12-06T16:08:29.665 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 3: abort() 2023-12-06T16:08:29.665 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x5594e07ed09f] 2023-12-06T16:08:29.665 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x5594e135a874] 2023-12-06T16:08:29.665 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5594e1365d11] 2023-12-06T16:08:29.665 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f646e3ea1ca] 2023-12-06T16:08:29.665 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 8: clone() 2023-12-06T16:08:29.665 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-06T16:08:29.665 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2023-12-06T16:08:29.673 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: -2529> 2023-12-06T16:08:29.638+0000 7f646256f700 -1 bdev(0x5594eb8d2400 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-06T16:08:29.674 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: -1803> 2023-12-06T16:08:29.640+0000 7f646256f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f646256f700 time 2023-12-06T16:08:29.639248+0000 2023-12-06T16:08:29.674 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-335-gd03fca4f/rpm/el8/BUILD/ceph-16.2.14-335-gd03fca4f/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-06T16:08:29.674 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2023-12-06T16:08:29.674 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:08:29.674 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x5594e07ecfce] 2023-12-06T16:08:29.674 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x5594e135a874] 2023-12-06T16:08:29.674 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5594e1365d11] 2023-12-06T16:08:29.675 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f646e3ea1ca] 2023-12-06T16:08:29.675 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 5: clone() 2023-12-06T16:08:29.675 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2023-12-06T16:08:29.675 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: -1001> 2023-12-06T16:08:29.642+0000 7f646256f700 -1 *** Caught signal (Aborted) ** 2023-12-06T16:08:29.675 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: in thread 7f646256f700 thread_name:bstore_aio 2023-12-06T16:08:29.675 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2023-12-06T16:08:29.675 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:08:29.675 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f646e3f4cf0] 2023-12-06T16:08:29.675 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2: gsignal() 2023-12-06T16:08:29.675 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 3: abort() 2023-12-06T16:08:29.675 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x5594e07ed09f] 2023-12-06T16:08:29.675 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x5594e135a874] 2023-12-06T16:08:29.676 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5594e1365d11] 2023-12-06T16:08:29.676 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f646e3ea1ca] 2023-12-06T16:08:29.676 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 8: clone() 2023-12-06T16:08:29.676 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-06T16:08:29.676 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr: 2023-12-06T16:08:29.842 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:daemon-helper: command failed with exit status 5 2023-12-06T16:08:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=124 2023-12-06T16:08:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2023-12-06T16:08:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 124 -eq 124 ']' 2023-12-06T16:08:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:08:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2023-12-06T16:08:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:08:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:08:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T16:08:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2023-12-06T16:08:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2023-12-06T16:08:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2023-12-06T16:08:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2023-12-06T16:08:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_10 2023-12-06T16:08:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:08:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:08:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_10 2023-12-06T16:08:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-06T16:08:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:08:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10ca31c4a5df 2023-12-06T16:08:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-06T16:08:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2023-12-06T16:08:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror image_10 present 10ca31c4a5df 2023-12-06T16:08:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2023-12-06T16:08:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:08:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T16:08:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2023-12-06T16:08:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10ca31c4a5df 2023-12-06T16:08:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2023-12-06T16:08:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2023-12-06T16:08:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror image_10 2023-12-06T16:08:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2023-12-06T16:08:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2023-12-06T16:08:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=image_10 2023-12-06T16:08:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-06T16:08:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-06T16:08:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10ca31c4a5df 2023-12-06T16:08:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10ca31c4a5df 2023-12-06T16:08:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10ca31c4a5df -o 10ca31c4a5df = 10ca31c4a5df 2023-12-06T16:08:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2023-12-06T16:08:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2023-12-06T16:08:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2023-12-06T16:08:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror image_10 2023-12-06T16:08:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2023-12-06T16:08:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2023-12-06T16:08:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=image_10 2023-12-06T16:08:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2023-12-06T16:08:34.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-12-06T16:08:34.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2023-12-06T16:08:34.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:08:34.977+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:08:39.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:08:39.978+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:08:40.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-12-06T16:08:40.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~5s 2023-12-06T16:08:44.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:08:44.977+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:08:45.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-12-06T16:08:45.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~11s 2023-12-06T16:08:49.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:08:49.979+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:08:50.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-12-06T16:08:50.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~16s 2023-12-06T16:08:54.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:08:54.980+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:08:55.990 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~194s 2023-12-06T16:08:55.991 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~21s 2023-12-06T16:08:59.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:08:59.981+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:09:01.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~199s 2023-12-06T16:09:01.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~27s 2023-12-06T16:09:04.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:04.983+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:09:06.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-12-06T16:09:06.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~32s 2023-12-06T16:09:09.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:09.983+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:09:11.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~210s 2023-12-06T16:09:11.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~37s 2023-12-06T16:09:14.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:14.984+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:09:17.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~215s 2023-12-06T16:09:17.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~42s 2023-12-06T16:09:19.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:19.985+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:09:22.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~221s 2023-12-06T16:09:22.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~48s 2023-12-06T16:09:24.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:24.986+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:09:25.310 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9352877122183839) > weight (0.25) 2023-12-06T16:09:25.310 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.34210879491597146) > weight (0.25) 2023-12-06T16:09:25.311 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.41415644161205334) > weight (0.25) 2023-12-06T16:09:25.311 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-06T16:09:25.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-06T16:09:25.311 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-06T16:09:25.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:25.311+0000 7fbf3fe37700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 119384) UID: 1000 2023-12-06T16:09:26.312 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-06T16:09:26.312 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T16:09:27.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~226s 2023-12-06T16:09:27.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~53s 2023-12-06T16:09:29.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:29.987+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:09:33.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~231s 2023-12-06T16:09:33.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~58s 2023-12-06T16:09:34.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:34.988+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:09:38.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~237s 2023-12-06T16:09:38.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~64s 2023-12-06T16:09:39.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:39.989+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:09:40.487 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6032867106634296) > weight (0.25) 2023-12-06T16:09:40.487 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9430373188069151) > weight (0.25) 2023-12-06T16:09:40.487 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6174635251493494) > weight (0.25) 2023-12-06T16:09:40.487 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4431554734290679) > weight (0.25) 2023-12-06T16:09:40.487 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2023-12-06T16:09:43.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~242s 2023-12-06T16:09:43.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~69s 2023-12-06T16:09:44.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:44.990+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:09:49.025 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~247s 2023-12-06T16:09:49.025 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~74s 2023-12-06T16:09:49.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:49.991+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:09:54.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~252s 2023-12-06T16:09:54.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~80s 2023-12-06T16:09:54.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:54.992+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:09:59.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~258s 2023-12-06T16:09:59.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~85s 2023-12-06T16:09:59.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:09:59.993+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:10:04.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~263s 2023-12-06T16:10:04.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~90s 2023-12-06T16:10:04.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:10:04.994+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:10:09.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:10:09.995+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:10:10.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~268s 2023-12-06T16:10:10.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~95s 2023-12-06T16:10:14.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:10:14.996+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:10:15.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~274s 2023-12-06T16:10:15.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~101s 2023-12-06T16:10:19.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:10:19.997+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:10:20.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~279s 2023-12-06T16:10:20.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~106s 2023-12-06T16:10:24.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:10:24.998+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:10:26.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~284s 2023-12-06T16:10:26.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~111s 2023-12-06T16:10:30.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:10:29.999+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:10:31.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~290s 2023-12-06T16:10:31.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~117s 2023-12-06T16:10:35.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:10:35.000+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:10:36.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~295s 2023-12-06T16:10:36.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~122s 2023-12-06T16:10:40.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:10:40.001+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:10:42.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~300s 2023-12-06T16:10:42.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~127s 2023-12-06T16:10:42.059 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-06T16:10:42.362 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-12-06T16:10:42.363 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-12-06T16:10:42.363 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-12-06T16:10:42.364 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T16:10:42.363+0000 7f44bd44a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 115488) UID: 0 2023-12-06T16:10:42.364 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T16:10:42.363+0000 7f44bd44a700 -1 osd.2 494 *** Got signal Terminated *** 2023-12-06T16:10:42.364 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2023-12-06T16:10:42.363+0000 7f44bd44a700 -1 osd.2 494 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-06T16:10:42.364 INFO:tasks.ceph.cluster2.mon.a.smithi204.stderr:2023-12-06T16:10:42.363+0000 7fe8cc4ed700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 115112) UID: 0 2023-12-06T16:10:42.364 INFO:tasks.ceph.cluster2.mon.a.smithi204.stderr:2023-12-06T16:10:42.363+0000 7fe8cc4ed700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-06T16:10:42.365 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T16:10:42.364+0000 7fa47bada700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 118394) UID: 0 2023-12-06T16:10:42.365 INFO:tasks.ceph.cluster2.mgr.x.smithi176.stderr:2023-12-06T16:10:42.364+0000 7fa47bada700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-06T16:10:45.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:10:45.003+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:10:50.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:10:50.003+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:10:51.489 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6809148886233819) > weight (0.25) 2023-12-06T16:10:51.489 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5155563006466211) > weight (0.25) 2023-12-06T16:10:51.489 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3732705392938398) > weight (0.25) 2023-12-06T16:10:51.489 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4086361256316575) > weight (0.25) 2023-12-06T16:10:51.489 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2023-12-06T16:10:55.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:10:55.003+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:11:00.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:11:00.004+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:11:05.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:11:05.005+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:11:10.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:11:10.007+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:11:15.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:11:15.008+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:11:20.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:11:20.009+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:11:25.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:11:25.010+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:11:30.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:11:30.011+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:11:35.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:11:35.012+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:11:40.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:11:40.013+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:11:45.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:11:45.014+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:11:50.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:11:50.015+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:11:55.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:11:55.016+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:12:00.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:12:00.015+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:12:05.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:12:05.016+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:12:10.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:12:10.017+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:12:15.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:12:15.018+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:12:20.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:12:20.019+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:12:25.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:12:25.020+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:12:30.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:12:30.021+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:12:35.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:12:35.022+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:12:36.490 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9609684587186867) > weight (0.25) 2023-12-06T16:12:36.490 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8634545643182944) > weight (0.25) 2023-12-06T16:12:36.490 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.27562863222305223) > weight (0.25) 2023-12-06T16:12:36.490 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4410058174797258) > weight (0.25) 2023-12-06T16:12:36.490 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2023-12-06T16:12:40.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:12:40.023+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:12:45.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:12:45.024+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:12:50.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:12:50.025+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:12:55.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:12:55.026+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:13:00.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:13:00.027+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:13:05.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:13:05.028+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:13:10.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:13:10.029+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:13:15.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:13:15.030+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:13:20.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:13:20.031+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:13:25.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:13:25.032+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:13:30.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:13:30.033+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:13:35.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:13:35.034+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:13:40.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:13:40.034+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:13:45.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:13:45.036+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:13:50.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:13:50.037+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:13:55.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:13:55.038+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:14:00.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:14:00.039+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:14:05.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:14:05.038+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:14:10.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:14:10.040+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:14:10.490 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.83025621571348) > weight (0.25) 2023-12-06T16:14:10.491 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.405060417085239) > weight (0.25) 2023-12-06T16:14:10.491 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5176385153964019) > weight (0.25) 2023-12-06T16:14:10.491 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-06T16:14:10.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-06T16:14:10.491 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-06T16:14:10.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr:2023-12-06T16:14:10.491+0000 7f935c9de700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 127069) UID: 1000 2023-12-06T16:14:13.493 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-06T16:14:13.493 INFO:teuthology.orchestra.run:waiting for 600 2023-12-06T16:14:15.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:14:15.040+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:14:20.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:14:20.041+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:14:25.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:14:25.042+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:14:30.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:14:30.043+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:14:35.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:14:35.044+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:14:40.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:14:40.045+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:14:45.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:14:45.046+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:14:50.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:14:50.047+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:14:55.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:14:55.048+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:15:00.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:15:00.049+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:15:05.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:15:05.049+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:15:10.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:15:10.050+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:15:15.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:15:15.050+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:15:20.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:15:20.051+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:15:25.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:15:25.053+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:15:30.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:15:30.054+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:15:35.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:15:35.055+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:15:40.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:15:40.056+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:15:45.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:15:45.057+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:15:50.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:15:50.058+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:15:55.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:15:55.059+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:16:00.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:16:00.060+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:16:05.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:16:05.061+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:16:10.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:16:10.062+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:16:15.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:16:15.062+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:16:20.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:16:20.062+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:16:25.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:16:25.064+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:16:30.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:16:30.065+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:16:35.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:16:35.066+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:16:40.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:16:40.067+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:16:45.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:16:45.068+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:16:50.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:16:50.068+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:16:55.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:16:55.069+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:17:00.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:17:00.070+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:17:05.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:17:05.071+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:17:10.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:17:10.072+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:17:15.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:17:15.073+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:17:20.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:17:20.073+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:17:25.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:17:25.074+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:17:30.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:17:30.075+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:17:35.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:17:35.077+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:17:40.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:17:40.078+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:17:45.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:17:45.078+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:17:50.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:17:50.079+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:17:55.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:17:55.080+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:18:00.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:18:00.081+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:18:05.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:18:05.082+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:18:10.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:18:10.083+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:18:15.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:18:15.084+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:18:20.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:18:20.084+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:18:25.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:18:25.086+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:18:30.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:18:30.087+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:18:35.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:18:35.088+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:18:40.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:18:40.089+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:18:45.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:18:45.090+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:18:50.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:18:50.091+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:18:55.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:18:55.092+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:00.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:19:00.093+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:05.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:19:05.094+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:10.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:19:10.095+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:15.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:19:15.096+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:20.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:19:20.096+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:25.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:19:25.097+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:19:30.098+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.453 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster1.client.mirror.3 2023-12-06T16:19:30.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 6 2023-12-06T16:19:30.454 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-06T16:19:30.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:*** Caught signal (Aborted) ** 2023-12-06T16:19:30.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: in thread 7fbf510986c0 thread_name:rbd-mirror 2023-12-06T16:19:30.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:19:30.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fbf46982cf0] 2023-12-06T16:19:30.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 2: /lib64/libpthread.so.0(+0x96cd) [0x7fbf469796cd] 2023-12-06T16:19:30.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 3: (Thread::join(void**)+0x31) [0x7fbf47633a11] 2023-12-06T16:19:30.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 4: (rbd::mirror::PoolReplayer::shut_down()+0x28a) [0x5575baa61b5a] 2023-12-06T16:19:30.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 5: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5575baa61bb2] 2023-12-06T16:19:30.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 6: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x5575baa59791] 2023-12-06T16:19:30.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 7: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5575baa53645] 2023-12-06T16:19:30.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 8: main() 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 9: __libc_start_main() 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 10: _start() 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:2023-12-06T16:19:30.455+0000 7fbf510986c0 -1 *** Caught signal (Aborted) ** 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: in thread 7fbf510986c0 thread_name:rbd-mirror 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fbf46982cf0] 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 2: /lib64/libpthread.so.0(+0x96cd) [0x7fbf469796cd] 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 3: (Thread::join(void**)+0x31) [0x7fbf47633a11] 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 4: (rbd::mirror::PoolReplayer::shut_down()+0x28a) [0x5575baa61b5a] 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 5: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5575baa61bb2] 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 6: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x5575baa59791] 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 7: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5575baa53645] 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 8: main() 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 9: __libc_start_main() 2023-12-06T16:19:30.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 10: _start() 2023-12-06T16:19:30.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-06T16:19:30.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 2023-12-06T16:19:30.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -9051> 2023-12-06T16:06:19.514+0000 7fbf3261c700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c186c400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2023-12-06T16:19:30.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -9050> 2023-12-06T16:06:19.514+0000 7fbf3261c700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c186c400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2023-12-06T16:19:30.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -7998> 2023-12-06T16:06:59.590+0000 7fbf3261c700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c16ed000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2023-12-06T16:19:30.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -7997> 2023-12-06T16:06:59.590+0000 7fbf3261c700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c16ed000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2023-12-06T16:19:30.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -6815> 2023-12-06T16:07:39.710+0000 7fbf33e1f700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c17d4400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2023-12-06T16:19:30.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -6814> 2023-12-06T16:07:39.710+0000 7fbf33e1f700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c17d4400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2023-12-06T16:19:30.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -5684> 2023-12-06T16:08:25.853+0000 7fbf33e1f700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c1a24c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2023-12-06T16:19:30.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -5683> 2023-12-06T16:08:25.853+0000 7fbf33e1f700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5575c1a24c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=da654ae0-0569-4aa4-8355-100f87001bea, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2023-12-06T16:19:30.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -5075> 2023-12-06T16:08:34.977+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4984> 2023-12-06T16:08:39.978+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4949> 2023-12-06T16:08:44.977+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4905> 2023-12-06T16:08:49.979+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4870> 2023-12-06T16:08:54.980+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4826> 2023-12-06T16:08:59.981+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4735> 2023-12-06T16:09:04.983+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4644> 2023-12-06T16:09:09.983+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4609> 2023-12-06T16:09:14.984+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4565> 2023-12-06T16:09:19.985+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4530> 2023-12-06T16:09:24.986+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4523> 2023-12-06T16:09:25.311+0000 7fbf3fe37700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 119384) UID: 1000 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4351> 2023-12-06T16:09:29.987+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4298> 2023-12-06T16:09:34.988+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4252> 2023-12-06T16:09:39.989+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4225> 2023-12-06T16:09:44.990+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4189> 2023-12-06T16:09:49.991+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4162> 2023-12-06T16:09:54.992+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4126> 2023-12-06T16:09:59.993+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4073> 2023-12-06T16:10:04.994+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4027> 2023-12-06T16:10:09.995+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -4000> 2023-12-06T16:10:14.996+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3964> 2023-12-06T16:10:19.997+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3937> 2023-12-06T16:10:24.998+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3901> 2023-12-06T16:10:29.999+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3848> 2023-12-06T16:10:35.000+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3802> 2023-12-06T16:10:40.001+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3775> 2023-12-06T16:10:45.003+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3739> 2023-12-06T16:10:50.003+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3712> 2023-12-06T16:10:55.003+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3676> 2023-12-06T16:11:00.004+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3621> 2023-12-06T16:11:05.005+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3583> 2023-12-06T16:11:10.007+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3556> 2023-12-06T16:11:15.008+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3518> 2023-12-06T16:11:20.009+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3491> 2023-12-06T16:11:25.010+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3455> 2023-12-06T16:11:30.011+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3404> 2023-12-06T16:11:35.012+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3368> 2023-12-06T16:11:40.013+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3341> 2023-12-06T16:11:45.014+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3305> 2023-12-06T16:11:50.015+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3278> 2023-12-06T16:11:55.016+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3242> 2023-12-06T16:12:00.015+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3191> 2023-12-06T16:12:05.016+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3155> 2023-12-06T16:12:10.017+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3128> 2023-12-06T16:12:15.018+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3092> 2023-12-06T16:12:20.019+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3065> 2023-12-06T16:12:25.020+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -3004> 2023-12-06T16:12:30.021+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2953> 2023-12-06T16:12:35.022+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2917> 2023-12-06T16:12:40.023+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2890> 2023-12-06T16:12:45.024+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2854> 2023-12-06T16:12:50.025+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2827> 2023-12-06T16:12:55.026+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2791> 2023-12-06T16:13:00.027+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2736> 2023-12-06T16:13:05.028+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2700> 2023-12-06T16:13:10.029+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2673> 2023-12-06T16:13:15.030+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2637> 2023-12-06T16:13:20.031+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2610> 2023-12-06T16:13:25.032+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2574> 2023-12-06T16:13:30.033+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2523> 2023-12-06T16:13:35.034+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2487> 2023-12-06T16:13:40.034+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2460> 2023-12-06T16:13:45.036+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2424> 2023-12-06T16:13:50.037+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2397> 2023-12-06T16:13:55.038+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2361> 2023-12-06T16:14:00.039+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2307> 2023-12-06T16:14:05.038+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2271> 2023-12-06T16:14:10.040+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2244> 2023-12-06T16:14:15.040+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2208> 2023-12-06T16:14:20.041+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2181> 2023-12-06T16:14:25.042+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2145> 2023-12-06T16:14:30.043+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2094> 2023-12-06T16:14:35.044+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2058> 2023-12-06T16:14:40.045+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -2031> 2023-12-06T16:14:45.046+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1995> 2023-12-06T16:14:50.047+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1968> 2023-12-06T16:14:55.048+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1932> 2023-12-06T16:15:00.049+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1881> 2023-12-06T16:15:05.049+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1845> 2023-12-06T16:15:10.050+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1818> 2023-12-06T16:15:15.050+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1782> 2023-12-06T16:15:20.051+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1755> 2023-12-06T16:15:25.053+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1719> 2023-12-06T16:15:30.054+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1668> 2023-12-06T16:15:35.055+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1632> 2023-12-06T16:15:40.056+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1605> 2023-12-06T16:15:45.057+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1569> 2023-12-06T16:15:50.058+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1542> 2023-12-06T16:15:55.059+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1506> 2023-12-06T16:16:00.060+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1455> 2023-12-06T16:16:05.061+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1419> 2023-12-06T16:16:10.062+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1392> 2023-12-06T16:16:15.062+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1356> 2023-12-06T16:16:20.062+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1329> 2023-12-06T16:16:25.064+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1293> 2023-12-06T16:16:30.065+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1242> 2023-12-06T16:16:35.066+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1206> 2023-12-06T16:16:40.067+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1179> 2023-12-06T16:16:45.068+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1143> 2023-12-06T16:16:50.068+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1116> 2023-12-06T16:16:55.069+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1080> 2023-12-06T16:17:00.070+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -1021> 2023-12-06T16:17:05.071+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -985> 2023-12-06T16:17:10.072+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -958> 2023-12-06T16:17:15.073+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -922> 2023-12-06T16:17:20.073+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -895> 2023-12-06T16:17:25.074+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -859> 2023-12-06T16:17:30.075+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -808> 2023-12-06T16:17:35.077+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -772> 2023-12-06T16:17:40.078+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -745> 2023-12-06T16:17:45.078+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -709> 2023-12-06T16:17:50.079+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -682> 2023-12-06T16:17:55.080+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -646> 2023-12-06T16:18:00.081+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -595> 2023-12-06T16:18:05.082+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -559> 2023-12-06T16:18:10.083+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -532> 2023-12-06T16:18:15.084+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -496> 2023-12-06T16:18:20.084+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -469> 2023-12-06T16:18:25.086+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -433> 2023-12-06T16:18:30.087+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -382> 2023-12-06T16:18:35.088+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -346> 2023-12-06T16:18:40.089+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -319> 2023-12-06T16:18:45.090+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -283> 2023-12-06T16:18:50.091+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -256> 2023-12-06T16:18:55.092+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -220> 2023-12-06T16:19:00.093+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -169> 2023-12-06T16:19:05.094+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -133> 2023-12-06T16:19:10.095+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -106> 2023-12-06T16:19:15.096+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -70> 2023-12-06T16:19:20.096+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -43> 2023-12-06T16:19:25.097+0000 7fbf3ee35700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: -7> 2023-12-06T16:19:30.098+0000 7fbf3f636700 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x5575c186fec0 finish: resending after timeout 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 0> 2023-12-06T16:19:30.455+0000 7fbf510986c0 -1 *** Caught signal (Aborted) ** 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: in thread 7fbf510986c0 thread_name:rbd-mirror 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fbf46982cf0] 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 2: /lib64/libpthread.so.0(+0x96cd) [0x7fbf469796cd] 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 3: (Thread::join(void**)+0x31) [0x7fbf47633a11] 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 4: (rbd::mirror::PoolReplayer::shut_down()+0x28a) [0x5575baa61b5a] 2023-12-06T16:19:30.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 5: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5575baa61bb2] 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 6: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x5575baa59791] 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 7: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5575baa53645] 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 8: main() 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 9: __libc_start_main() 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 10: _start() 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:*** Caught signal (Segmentation fault) ** 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: in thread 7fbf510986c0 thread_name:rbd-mirror 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fbf46982cf0] 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 2: pthread_getname_np() 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 3: (ceph::logging::Log::dump_recent()+0x4c2) [0x7fbf478ccbb2] 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 4: rbd-mirror(+0xc0507b) [0x5575bb16907b] 2023-12-06T16:19:30.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 5: /lib64/libpthread.so.0(+0x12cf0) [0x7fbf46982cf0] 2023-12-06T16:19:30.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 6: /lib64/libpthread.so.0(+0x96cd) [0x7fbf469796cd] 2023-12-06T16:19:30.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 7: (Thread::join(void**)+0x31) [0x7fbf47633a11] 2023-12-06T16:19:30.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 8: (rbd::mirror::PoolReplayer::shut_down()+0x28a) [0x5575baa61b5a] 2023-12-06T16:19:30.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 9: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5575baa61bb2] 2023-12-06T16:19:30.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 10: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x5575baa59791] 2023-12-06T16:19:30.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 11: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5575baa53645] 2023-12-06T16:19:30.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 12: main() 2023-12-06T16:19:30.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 13: __libc_start_main() 2023-12-06T16:19:30.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr: 14: _start() 2023-12-06T16:19:30.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi204.stderr:daemon-helper: command crashed with signal 11 2023-12-06T16:19:35.619 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2023-12-06T16:19:35.620 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-06T16:19:36.123 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2023-12-06T16:19:36.123 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2023-12-06T16:19:36.124 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2023-12-06T16:19:36.124 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2023-12-06T16:19:36.124 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2023-12-06T16:19:36.125 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f0773ad9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 114674) UID: 0 2023-12-06T16:19:36.125 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f0773ad9700 -1 osd.0 457 *** Got signal Terminated *** 2023-12-06T16:19:36.125 INFO:tasks.ceph.cluster1.osd.0.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f0773ad9700 -1 osd.0 457 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-06T16:19:36.125 INFO:tasks.ceph.cluster1.osd.1.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f5a14aae700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 114675) UID: 0 2023-12-06T16:19:36.125 INFO:tasks.ceph.cluster1.osd.1.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f5a14aae700 -1 osd.1 457 *** Got signal Terminated *** 2023-12-06T16:19:36.126 INFO:tasks.ceph.cluster1.osd.1.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f5a14aae700 -1 osd.1 457 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-06T16:19:36.126 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f523a052700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 114680) UID: 0 2023-12-06T16:19:36.126 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f523a052700 -1 osd.2 457 *** Got signal Terminated *** 2023-12-06T16:19:36.126 INFO:tasks.ceph.cluster1.osd.2.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f523a052700 -1 osd.2 457 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-06T16:19:36.126 INFO:tasks.ceph.cluster1.mon.a.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f77252f9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 114271) UID: 0 2023-12-06T16:19:36.127 INFO:tasks.ceph.cluster1.mon.a.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f77252f9700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-06T16:19:36.127 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f7b91eb1700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 114273) UID: 0 2023-12-06T16:19:36.127 INFO:tasks.ceph.cluster1.mgr.x.smithi176.stderr:2023-12-06T16:19:36.124+0000 7f7b91eb1700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-06T16:24:17.640 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.3 2023-12-06T16:24:17.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 6 2023-12-06T16:24:17.641 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-06T16:24:17.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr:*** Caught signal (Aborted) ** 2023-12-06T16:24:17.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: in thread 7f936dc3f6c0 thread_name:rbd-mirror 2023-12-06T16:24:17.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:24:17.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f9363529cf0] 2023-12-06T16:24:17.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 2: pthread_cond_wait() 2023-12-06T16:24:17.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f936232b860] 2023-12-06T16:24:17.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 4: /lib64/librados.so.2(+0xa3a7b) [0x7f936d719a7b] 2023-12-06T16:24:17.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 5: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x62) [0x7f936d6f0fa2] 2023-12-06T16:24:17.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 6: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0x96) [0x55bc59a60ff6] 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 7: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x5db) [0x55bc593ba9db] 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 8: (rbd::mirror::ClusterWatcher::refresh_pools()+0x22d) [0x55bc593bc82d] 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 9: (rbd::mirror::Mirror::run()+0x1cd) [0x55bc59384c3d] 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 10: main() 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 11: __libc_start_main() 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 12: _start() 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr:2023-12-06T16:24:17.643+0000 7f936dc3f6c0 -1 *** Caught signal (Aborted) ** 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: in thread 7f936dc3f6c0 thread_name:rbd-mirror 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f9363529cf0] 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 2: pthread_cond_wait() 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f936232b860] 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 4: /lib64/librados.so.2(+0xa3a7b) [0x7f936d719a7b] 2023-12-06T16:24:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 5: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x62) [0x7f936d6f0fa2] 2023-12-06T16:24:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 6: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0x96) [0x55bc59a60ff6] 2023-12-06T16:24:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 7: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x5db) [0x55bc593ba9db] 2023-12-06T16:24:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 8: (rbd::mirror::ClusterWatcher::refresh_pools()+0x22d) [0x55bc593bc82d] 2023-12-06T16:24:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 9: (rbd::mirror::Mirror::run()+0x1cd) [0x55bc59384c3d] 2023-12-06T16:24:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 10: main() 2023-12-06T16:24:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 11: __libc_start_main() 2023-12-06T16:24:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 12: _start() 2023-12-06T16:24:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-06T16:24:17.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 2023-12-06T16:24:17.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: -1446> 2023-12-06T16:14:10.491+0000 7f935c9de700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 127069) UID: 1000 2023-12-06T16:24:17.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 0> 2023-12-06T16:24:17.643+0000 7f936dc3f6c0 -1 *** Caught signal (Aborted) ** 2023-12-06T16:24:17.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: in thread 7f936dc3f6c0 thread_name:rbd-mirror 2023-12-06T16:24:17.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f9363529cf0] 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 2: pthread_cond_wait() 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f936232b860] 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 4: /lib64/librados.so.2(+0xa3a7b) [0x7f936d719a7b] 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 5: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x62) [0x7f936d6f0fa2] 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 6: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0x96) [0x55bc59a60ff6] 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 7: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x5db) [0x55bc593ba9db] 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 8: (rbd::mirror::ClusterWatcher::refresh_pools()+0x22d) [0x55bc593bc82d] 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 9: (rbd::mirror::Mirror::run()+0x1cd) [0x55bc59384c3d] 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 10: main() 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 11: __libc_start_main() 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 12: _start() 2023-12-06T16:24:17.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-06T16:24:17.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 2023-12-06T16:24:17.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: -9999> 2023-12-06T16:14:10.491+0000 7f935c9de700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 127069) UID: 1000 2023-12-06T16:24:17.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: -9998> 2023-12-06T16:24:17.643+0000 7f936dc3f6c0 -1 *** Caught signal (Aborted) ** 2023-12-06T16:24:17.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: in thread 7f936dc3f6c0 thread_name:rbd-mirror 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: ceph version 16.2.14-335-gd03fca4f (d03fca4f35d43f61b4679936aef7bd2f01cced0b) pacific (stable) 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f9363529cf0] 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 2: pthread_cond_wait() 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f936232b860] 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 4: /lib64/librados.so.2(+0xa3a7b) [0x7f936d719a7b] 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 5: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x62) [0x7f936d6f0fa2] 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 6: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0x96) [0x55bc59a60ff6] 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 7: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x5db) [0x55bc593ba9db] 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 8: (rbd::mirror::ClusterWatcher::refresh_pools()+0x22d) [0x55bc593bc82d] 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 9: (rbd::mirror::Mirror::run()+0x1cd) [0x55bc59384c3d] 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 10: main() 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 11: __libc_start_main() 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 12: _start() 2023-12-06T16:24:17.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-06T16:24:17.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr: 2023-12-06T16:24:18.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi204.stderr:daemon-helper: command crashed with signal 6 2023-12-06T21:33:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Terminated 2023-12-06T21:33:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 status 2023-12-06T21:33:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cleanup 143 2023-12-06T21:33:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local error_code=143 2023-12-06T21:33:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set +e 2023-12-06T21:33:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 143 -ne 0 ']' 2023-12-06T21:33:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ status 2023-12-06T21:33:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster daemon image_pool image_ns image 2023-12-06T21:33:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for cluster in ${CLUSTER1} ${CLUSTER2} 2023-12-06T21:33:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 status' 2023-12-06T21:33:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-06T21:33:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --cluster cluster1 -s 2023-12-06T21:38:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2023-12-06T21:38:20.012+0000 7fcbf2749700 0 monclient(hunting): authenticate timed out after 300 2023-12-06T21:38:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-06T21:38:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-06T21:38:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --cluster cluster1 service dump 2023-12-06T21:43:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2023-12-06T21:43:20.114+0000 7f097c04d700 0 monclient(hunting): authenticate timed out after 300 2023-12-06T21:43:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-06T21:43:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-06T21:43:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --cluster cluster1 service status 2023-12-06T21:48:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2023-12-06T21:48:20.215+0000 7fd7746e4700 0 monclient(hunting): authenticate timed out after 300 2023-12-06T21:48:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-06T21:48:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T21:48:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T21:48:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-06T21:48:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-06T21:48:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror images 2023-12-06T21:48:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror images' 2023-12-06T21:48:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' ls -l 2023-12-06T21:53:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T21:53:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-06T21:53:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T21:53:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T21:53:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror mirror pool info 2023-12-06T21:53:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror mirror pool info' 2023-12-06T21:53:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' mirror pool info 2023-12-06T21:58:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T21:58:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T21:58:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror mirror pool status 2023-12-06T21:58:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T21:58:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror mirror pool status' 2023-12-06T21:58:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-06T21:58:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' mirror pool status --verbose 2023-12-06T22:03:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T22:03:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T22:03:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T22:03:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ rbd --cluster cluster1 -p mirror --namespace '' ls 2023-12-06T22:08:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror rbd_mirroring omap vals 2023-12-06T22:08:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror rbd_mirroring omap vals' 2023-12-06T22:08:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirroring 2023-12-06T22:13:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-06T22:13:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror rbd_mirror_leader omap vals' 2023-12-06T22:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror rbd_mirror_leader omap vals 2023-12-06T22:13:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirror_leader 2023-12-06T22:18:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-06T22:18:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T22:18:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T22:18:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-06T22:18:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror ns1 images' 2023-12-06T22:18:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror ns1 images 2023-12-06T22:18:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 ls -l 2023-12-06T22:23:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T22:23:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-06T22:23:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T22:23:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T22:23:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirrorns1 mirror pool info 2023-12-06T22:23:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirrorns1 mirror pool info' 2023-12-06T22:23:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool info 2023-12-06T22:28:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T22:28:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T22:28:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T22:28:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirrorns1 mirror pool status 2023-12-06T22:28:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirrorns1 mirror pool status' 2023-12-06T22:28:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-06T22:28:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool status --verbose 2023-12-06T22:33:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T22:33:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T22:33:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T22:33:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ rbd --cluster cluster1 -p mirror --namespace ns1 ls 2023-12-06T22:38:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror ns1 rbd_mirroring omap vals 2023-12-06T22:38:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror ns1 rbd_mirroring omap vals' 2023-12-06T22:38:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirroring 2023-12-06T22:43:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-06T22:43:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror ns1 rbd_mirror_leader omap vals 2023-12-06T22:43:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror ns1 rbd_mirror_leader omap vals' 2023-12-06T22:43:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2023-12-06T22:48:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-06T22:48:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T22:48:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T22:48:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-06T22:48:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror ns2 images 2023-12-06T22:48:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror ns2 images' 2023-12-06T22:48:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 ls -l 2023-12-06T22:53:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T22:53:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-06T22:53:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T22:53:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T22:53:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirrorns2 mirror pool info 2023-12-06T22:53:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirrorns2 mirror pool info' 2023-12-06T22:53:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool info 2023-12-06T22:58:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T22:58:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T22:58:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T22:58:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirrorns2 mirror pool status' 2023-12-06T22:58:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirrorns2 mirror pool status 2023-12-06T22:58:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-06T22:58:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool status --verbose 2023-12-06T23:03:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T23:03:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T23:03:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T23:03:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ rbd --cluster cluster1 -p mirror --namespace ns2 ls 2023-12-06T23:08:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror ns2 rbd_mirroring omap vals 2023-12-06T23:08:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror ns2 rbd_mirroring omap vals' 2023-12-06T23:08:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirroring 2023-12-06T23:13:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-06T23:13:20.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror ns2 rbd_mirror_leader omap vals' 2023-12-06T23:13:20.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror ns2 rbd_mirror_leader omap vals 2023-12-06T23:13:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2023-12-06T23:18:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-06T23:18:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T23:18:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T23:18:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-06T23:18:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-06T23:18:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parent images 2023-12-06T23:18:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parent images' 2023-12-06T23:18:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' ls -l 2023-12-06T23:23:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T23:23:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-06T23:23:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T23:23:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parent mirror pool info 2023-12-06T23:23:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T23:23:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parent mirror pool info' 2023-12-06T23:23:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool info 2023-12-06T23:28:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T23:28:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T23:28:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T23:28:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parent mirror pool status 2023-12-06T23:28:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parent mirror pool status' 2023-12-06T23:28:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-06T23:28:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool status --verbose 2023-12-06T23:33:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T23:33:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T23:33:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T23:33:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ rbd --cluster cluster1 -p mirror_parent --namespace '' ls 2023-12-06T23:38:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parent rbd_mirroring omap vals 2023-12-06T23:38:20.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parent rbd_mirroring omap vals' 2023-12-06T23:38:20.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2023-12-06T23:43:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-06T23:43:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parent rbd_mirror_leader omap vals' 2023-12-06T23:43:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parent rbd_mirror_leader omap vals 2023-12-06T23:43:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2023-12-06T23:48:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-06T23:48:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T23:48:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T23:48:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-06T23:48:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parent ns1 images 2023-12-06T23:48:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parent ns1 images' 2023-12-06T23:48:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls -l 2023-12-06T23:53:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T23:53:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-06T23:53:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T23:53:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T23:53:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parentns1 mirror pool info 2023-12-06T23:53:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parentns1 mirror pool info' 2023-12-06T23:53:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool info 2023-12-06T23:58:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-06T23:58:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-06T23:58:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parentns1 mirror pool status 2023-12-06T23:58:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-06T23:58:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parentns1 mirror pool status' 2023-12-06T23:58:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-06T23:58:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool status --verbose 2023-12-07T00:03:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T00:03:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T00:03:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T00:03:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls 2023-12-07T00:08:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parent ns1 rbd_mirroring omap vals 2023-12-07T00:08:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parent ns1 rbd_mirroring omap vals' 2023-12-07T00:08:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2023-12-07T00:13:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-07T00:13:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parent ns1 rbd_mirror_leader omap vals 2023-12-07T00:13:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parent ns1 rbd_mirror_leader omap vals' 2023-12-07T00:13:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2023-12-07T00:18:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-07T00:18:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T00:18:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T00:18:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-07T00:18:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parent ns2 images 2023-12-07T00:18:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parent ns2 images' 2023-12-07T00:18:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls -l 2023-12-07T00:23:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T00:23:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-07T00:23:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T00:23:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parentns2 mirror pool info 2023-12-07T00:23:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T00:23:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parentns2 mirror pool info' 2023-12-07T00:23:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool info 2023-12-07T00:28:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T00:28:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T00:28:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T00:28:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parentns2 mirror pool status 2023-12-07T00:28:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parentns2 mirror pool status' 2023-12-07T00:28:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-07T00:28:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool status --verbose 2023-12-07T00:33:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T00:33:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T00:33:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T00:33:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls 2023-12-07T00:38:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parent ns2 rbd_mirroring omap vals 2023-12-07T00:38:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parent ns2 rbd_mirroring omap vals' 2023-12-07T00:38:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2023-12-07T00:43:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-07T00:43:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster1 mirror_parent ns2 rbd_mirror_leader omap vals' 2023-12-07T00:43:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster1 mirror_parent ns2 rbd_mirror_leader omap vals 2023-12-07T00:43:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2023-12-07T00:48:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-07T00:48:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T00:48:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for cluster in ${CLUSTER1} ${CLUSTER2} 2023-12-07T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 status' 2023-12-07T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 status 2023-12-07T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-07T00:48:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --cluster cluster2 -s 2023-12-07T00:53:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2023-12-07T00:53:21.307+0000 7f6050557700 0 monclient(hunting): authenticate timed out after 300 2023-12-07T00:53:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-07T00:53:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-07T00:53:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --cluster cluster2 service dump 2023-12-07T00:58:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2023-12-07T00:58:21.407+0000 7eff98aa7700 0 monclient(hunting): authenticate timed out after 300 2023-12-07T00:58:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-07T00:58:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-07T00:58:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --cluster cluster2 service status 2023-12-07T01:03:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2023-12-07T01:03:21.508+0000 7fe7ee3cf700 0 monclient(hunting): authenticate timed out after 300 2023-12-07T01:03:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-07T01:03:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T01:03:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T01:03:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-07T01:03:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-07T01:03:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror images 2023-12-07T01:03:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror images' 2023-12-07T01:03:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' ls -l 2023-12-07T01:08:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T01:08:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-07T01:08:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T01:08:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T01:08:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror mirror pool info 2023-12-07T01:08:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror mirror pool info' 2023-12-07T01:08:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' mirror pool info 2023-12-07T01:13:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T01:13:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T01:13:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T01:13:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror mirror pool status 2023-12-07T01:13:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror mirror pool status' 2023-12-07T01:13:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-07T01:13:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' mirror pool status --verbose 2023-12-07T01:18:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T01:18:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T01:18:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T01:18:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ rbd --cluster cluster2 -p mirror --namespace '' ls 2023-12-07T01:23:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror rbd_mirroring omap vals 2023-12-07T01:23:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror rbd_mirroring omap vals' 2023-12-07T01:23:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirroring 2023-12-07T01:28:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-07T01:28:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror rbd_mirror_leader omap vals 2023-12-07T01:28:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror rbd_mirror_leader omap vals' 2023-12-07T01:28:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirror_leader 2023-12-07T01:33:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-07T01:33:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T01:33:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T01:33:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-07T01:33:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror ns1 images' 2023-12-07T01:33:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror ns1 images 2023-12-07T01:33:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 ls -l 2023-12-07T01:38:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T01:38:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-07T01:38:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T01:38:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T01:38:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirrorns1 mirror pool info 2023-12-07T01:38:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirrorns1 mirror pool info' 2023-12-07T01:38:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool info 2023-12-07T01:43:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T01:43:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T01:43:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirrorns1 mirror pool status 2023-12-07T01:43:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T01:43:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirrorns1 mirror pool status' 2023-12-07T01:43:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-07T01:43:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool status --verbose 2023-12-07T01:48:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T01:48:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T01:48:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T01:48:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ rbd --cluster cluster2 -p mirror --namespace ns1 ls 2023-12-07T01:53:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror ns1 rbd_mirroring omap vals 2023-12-07T01:53:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror ns1 rbd_mirroring omap vals' 2023-12-07T01:53:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirroring 2023-12-07T01:58:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-07T01:58:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror ns1 rbd_mirror_leader omap vals 2023-12-07T01:58:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror ns1 rbd_mirror_leader omap vals' 2023-12-07T01:58:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2023-12-07T02:03:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-07T02:03:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T02:03:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T02:03:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-07T02:03:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror ns2 images 2023-12-07T02:03:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror ns2 images' 2023-12-07T02:03:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 ls -l 2023-12-07T02:08:21.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T02:08:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-07T02:08:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T02:08:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T02:08:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirrorns2 mirror pool info 2023-12-07T02:08:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirrorns2 mirror pool info' 2023-12-07T02:08:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool info 2023-12-07T02:13:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T02:13:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T02:13:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T02:13:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirrorns2 mirror pool status 2023-12-07T02:13:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirrorns2 mirror pool status' 2023-12-07T02:13:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-07T02:13:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool status --verbose 2023-12-07T02:18:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T02:18:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T02:18:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T02:18:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ rbd --cluster cluster2 -p mirror --namespace ns2 ls 2023-12-07T02:23:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror ns2 rbd_mirroring omap vals 2023-12-07T02:23:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror ns2 rbd_mirroring omap vals' 2023-12-07T02:23:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirroring 2023-12-07T02:28:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-07T02:28:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror ns2 rbd_mirror_leader omap vals 2023-12-07T02:28:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror ns2 rbd_mirror_leader omap vals' 2023-12-07T02:28:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2023-12-07T02:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-07T02:33:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T02:33:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T02:33:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-07T02:33:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-07T02:33:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror_parent images 2023-12-07T02:33:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror_parent images' 2023-12-07T02:33:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' ls -l 2023-12-07T02:38:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T02:38:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-07T02:38:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T02:38:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror_parent mirror pool info 2023-12-07T02:38:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T02:38:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror_parent mirror pool info' 2023-12-07T02:38:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool info 2023-12-07T02:43:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T02:43:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T02:43:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror_parent mirror pool status 2023-12-07T02:43:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T02:43:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror_parent mirror pool status' 2023-12-07T02:43:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-07T02:43:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool status --verbose 2023-12-07T02:48:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T02:48:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T02:48:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T02:48:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ rbd --cluster cluster2 -p mirror_parent --namespace '' ls 2023-12-07T02:53:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror_parent rbd_mirroring omap vals 2023-12-07T02:53:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror_parent rbd_mirroring omap vals' 2023-12-07T02:53:22.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2023-12-07T02:58:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-07T02:58:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror_parent rbd_mirror_leader omap vals' 2023-12-07T02:58:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror_parent rbd_mirror_leader omap vals 2023-12-07T02:58:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2023-12-07T03:03:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-07T03:03:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T03:03:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T03:03:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-07T03:03:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror_parent ns1 images' 2023-12-07T03:03:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror_parent ns1 images 2023-12-07T03:03:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l 2023-12-07T03:08:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T03:08:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-07T03:08:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T03:08:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T03:08:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror_parentns1 mirror pool info 2023-12-07T03:08:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror_parentns1 mirror pool info' 2023-12-07T03:08:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool info 2023-12-07T03:13:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: couldn't connect to the cluster! 2023-12-07T03:13:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 2023-12-07T03:13:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 2023-12-07T03:13:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cluster2 mirror_parentns1 mirror pool status 2023-12-07T03:13:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'cluster2 mirror_parentns1 mirror pool status' 2023-12-07T03:13:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2023-12-07T03:13:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool status --verbose 2023-12-07T03:17:06.290 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-12-07T03:17:06.291 DEBUG:teuthology.task.console_log:Killing console logger for smithi176 2023-12-07T03:17:06.292 DEBUG:teuthology.task.console_log:Killing console logger for smithi204 2023-12-07T03:17:06.292 DEBUG:teuthology.exit:Finished running handlers